Greg discusses Palantirs experience and growing pains in scaling their Apollo Platforms internal use of GitOps to 100+ environments, 500+ unique services and 1000+ engineers. Natan will go through both sides of the argument. This plugin will allow customers to integrate Protons infrastructure templating, provisioning, and lifecycle management into their Backstage portal to reduce platform engineering overhead. The cluster provisioner GitHub Action detects the `cluster.yaml` in a GitHub repository and uses the provided configuration to provision an EKS cluster. Lately, service catalogs have been coming up in conversations more and more, especially when we speak with mid or large size enterprise accounts. Most likely you are asking yourself this question because we described just the beginning of the infra development flow. To request features or improvements to the Proton plugin, you can raise issues in the project GitHub repository or take a look as the AWS Proton service public roadmap. More than that, we want to add all our concerns and standards during the creation process. Since Backstage and the community have made a large set of ready-to-use React components available it's even easier to build GUIs with a limited background in React. Niall is a Sr. View Argo CD status for your projects in Backstage. One of the great features of Backstage is Software Templates, the ability for platform engineers to define templates that development teams can use to bootstrap new projects. Pleasesuggesta new one. 2022, Amazon Web Services, Inc. or its affiliates. View GitHub Insights for your components in Backstage. Stay tuned! Google's Lighthouse tool is a great resource for benchmarking and improving the accessibility, performance, SEO, and best practices of your website. What are resources in Humanitec and how do I use them? The project is in the incubation stage in the Cloud Native Computing Foundation (CNCF), having moved out of beta with the 1.0 release in March 2022. Opsgenie offers a simple way to associate alerts to components and visualize incidents. This plugin will show you the cost and resource usage of your application within Google Kubernetes Engine (GKE). Create new software components in just a few steps, with your standards built-in (Scaffolder). CodeScene is a multi-purpose tool bridging code, business and people. Platform veteran Jon Skarpeteig will share his insights about design choices, trade-offs and challenges you are likely to encounter when designing and building your IDP. He covered how they manage changes, and how those changes make it to production. Write your docs right alongside your code. Although Backstage is built on React, it is relatively easy to understand, even for somebody like me who rarely writes front-end applications. The plugin provides two mechanisms to integrate AWS Proton and Backstage, which can be used together or separately as necessary: Lets take a look at each of these in more detail. To see all the open source plugins built by the Backstage community, visit backstage.io/plugins. Developer portals provide us with a unified UI for all the backend systems in an organization, helping unify the way developers and DevOps work. Write your docs right alongside your code. Architecture, scaling, mobile and web development, management and more, this publication aggregates blog posts written by our very own Wix engineers. Components to display code quality metrics from SonarCloud and SonarQube. Click here to return to Amazon Web Services homepage, Using the AWS Proton plugins for Backstage tutorial, An entity card that can be added to the component UI to provide an overview of the current state of the Proton service associated with a Backstage catalog component, The Backstage scaffolder will template the initial source code for the application and push it to a new GitHub repository, The action invokes the Proton CreateService API to create the corresponding service using a Proton service template that has already been created by the platform team, Proton will create the service and service instances associated with the specification it was provided in the template, The CodePipeline pipeline created by the Proton service will access the code in the GitHub repository to build and deploy the service to one or more service instances, Creating the pre-requisite AWS Proton resources, Bootstrapping your own Backstage application, Installing the Proton plugin for Backstage, Authoring a software template that integrates the plugin as part of the scaffolding process, Creating a component that uses this software template. The top 10 fallacies in platform engineering. Observability for Apache Kafka clusters and async API of components. Oops! To learn more, please read our blog post and join the private waitlist. I previously built a GitOps workflow for EKS with GitHub Actions and intended to write a user interface for it, however Im not a front-end developer. Elrick will introduce use cases and present a UX prototype to use as a starting point of the community discussion. You create a microservice/lib/package from a template using the UI at hand. CI/CD, local development, many programming languages, the list goes on and on. Terraform is the most adopted Infrastructure as Code tool in the market. However, I wanted the cluster bootstrapping process to be more GitOps-ish, so this is what I did. Ill elaborate on that in my next meetup in Wix engineering soon. If you too work in a large dev organization (>300 developers), this probably comes as no surprise. GoCD is an open-source tool which is used in software development to help teams and organizations automate the continuous delivery of software. Its consistently becoming harder to have a full picture of what service is running on which infrastructure, who operates it or who owns it. GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Join our community to receive invitations to events, recordings, and networking opportunities among other Backstage users. This allows developers to get coding right away, using a predefined service design and language framework like Golang, Node.js, etc. Jenkins offers a simple way to set up a continuous integration and continuous delivery environment. View FOSSA license compliance of your components in Backstage. We are very excited to host another session with Bryan Finster, founder and former lead of the Walmart DevOps Dojo. Easily run git CLI commands from your scaffolder actions, An action to fire an arbitrary HTTP request. Were looking forward to seeing how customers can combine Backstage and AWS Proton to power their IDP initiatives, and were excited to evolve this integration as the Backstage project moves forward. See what plugins are alreadyin progressand . Embed dashboards (like DORA metrics) in your team or service pages from any dev tools (including Github, Gitlab, Jira, Argo, CircleCI, Buildkite, Pagerduty, Rollbar, Sent View monitoring info from dynatrace for services in your software catalog. Finally, the service catalog allows Ops teams to know at a glance which versions of a particular service are used by which applications and in which environments. Create new software components in just a few steps with your standards built-in (Scaffolder). View Rollbar errors for your services in Backstage. Easily view your Azure Pipelines within the Software Catalog. It doesnt matter if your service is Nodejs, Golang, React component or a Terraform module you need a solution to easily create new components and to follow the company code standards. Hell talk about the problems encountered, unsuccessful attempts at solving them and ultimately Apollos reimagination of GitOps at Palantir. Zooming in, every service catalog should have some version of these four core elements. Under the hood the profile uses the command line interface, EKSctl with GitHub Actions to provision a Highly Available Kubernetes cluster that includes a preloaded profile within a few minutes. Building GUIs with Backstages plugin architecture is incredibly easy. Service catalogs like Spotifys Backstage are establishing themselves as the best answer to these issues. Splunk On-Call offers a simple way to identify incidents and escalation policies. The IDP provides a set of shared capabilities that provide a standardized way for development teams to deploy applications to production. Backstage is a framework that unifies the UX of your frequently used infrastructure tools and makes them available from a service catalogue. Now multiply it by each programming language youre using, and youll get lost. In this webinar, we will explain best practices for handling Terraform at scale. A lightweight API to help you build your Internal Developer Platform. Prioritize and reduce technical debt. Try to imagine that you have over 1000 NodeJs Developers in your company, and each one of them will create a new service from scratch. A critical part of a successful IDP is the ability for development teams to self-servewhether this be infrastructure provisioning, application deployments, or even kicking off new servicesreducing load on platform teams who can then reinvest in platform capabilities and standards rather than repetitive processes. Browse 150+ ways of integrating and managing your infrastructure with open source drivers. Most cloud providers tell you to set them, yet members of the original Kubernetes team at Google say not to use them! The new service comes with a predefined design and set of metadata, depending on the specifics of your Ops or Platform engineering team. Lets look at Terraform, the most common IaC for creating resources with several cloud providers. Join this webinar with Hila to learn more about best practices in incident management. Components to discover and display API entities as an extension to the catalog plugin. Periskop is a pull-based, language agnostic exception aggregator for microservice environments. Build, test, and deploy on Google's serverless CI/CD platform. They can also manage their own deployments, doing roll-backs and diffs, versioning configurations the same way they do with code in Git. View Snyk scanned vulnerabilities and license compliance of your components directly in Backstage. A docs-like-code solution to technical documentation. Load Backstage entities from AWS CloudFormation stacks. The project lowers the barrier of entry for organizations to build sophisticated developer portals with an extensible data model and integration capabilities. Plugin authors can contribute user interface elements that can used by platform teams to construct views that are most relevant to development teams. This lets platform teams define both application code templates, along with its associated Infrastructure-as-Code templates, presented together in a unified workflow for development teams to create new components and services. And all in a context where a central Platform team can set predefined rules and golden paths for all other app development teams to operate within. Import your Backstage entities into Rootly services and view incidents & analytics. Its simple sign up for our newsletter. Configuration GUI over GitOps using kpt, with WYSIWYG editing, review and approval, versioning and undo, and package cloning and upgrades. Cloudify provides a remote execution and environment management backend for Kubernetes, Terraform, Ansible, etc. However, even with Backstage providing the portal itself, platform teams must still invest significant time building out the ability to provision infrastructure and pipelines in a way that can be scaled across an organization, such as using Infrastructure-as-Code tooling like CloudFormation and Terraform. The shortcuts plugin allows a user to have easy access to pages within a Backstage app by storing them as "shortcuts" in the Sidebar. Please provide feedback on those issues and create new requests to give us a better idea of what youd like to see in Backstage. View Security Insights for your components in Backstage. An article explaining the approach is also available - https://frontside.com/blog/2022-03-24-testing-backstage-catalog-ingestors/. For information about Spotifys use of cookies, see our cookie policy. Integrates GraphiQL as a tool to browse GraphQL API endpoints inside Backstage. The plugin is called `gitops-profiles` which allows users to provision GitOps-managed clusters directly from Backstage. 2021 Spotify AB. View service incidents information from FireHydrant, such as active incidents and incident metrics, directly within Backstage. In the Kubernetes universe there is an extension API called. We also create backstage Chaos tests templates, Grafana dashboards etc, Everything that the DevOps need to write their infracode to be perfect. Browse TODO comments in your project's source code. View your cloud carbon footprint by estimating energy use (kilowatt-hours) and carbon emissions (metric tons CO2e) from public cloud usage. See all the plugins built by the Backstage community. Visualize, understand and optimize your team's cloud costs. If we think about it, writing the Terraform process code can be broken down into 2 phases: Lets say we want to automate the module creation process. Manage releases without having to juggle git commands. Users can click Explore GitOps Clusters to create and manage clusters using the GitOps plugin. We took those plugin capabilities of Backstage and developed on top of it a fully Terraform E2E flow which can be used to create infra and deploy it to the cloud. Track usage of your Backstage instance using Google Analytics. Today, were excited to announce that AWS will be joining the organizations contributing to the Backstage open source community with our AWS Proton plugin for Backstage. The badges plugin offers a set of badges that can be used outside of Backstage, showing information related to data from the catalog. These templates will typically seed a new source code repository with a basic skeleton application, perhaps also including related artifacts like a Dockerfile and CI/CD pipeline. Things like security as code or Finops tags, validations, etc. One of the most common types of interface element that a plugin might contribute is commonly referred to as an Entity Card, which is generally a small UI widget that can be included on the overview page of something like a component entity. The area of infrastructure provisioning is one that Backstage leaves to the platform teams to implement, with tools like CloudFormation and Terraform. Adding new functionality to the Backstage application is as simple as creating a plugin, defining the plugin url (for example, `/gitops-clusters`), wiring it up to the applications router, and then adding the plugin url to the sidebar. Track usage of your Backstage instance using Google Analytics. This is just the dev experience. Spotify open sourced parts of its internal project, Backstage, on GitHub. At Humanitec, we often get asked how this functionality compares to that of Internal Developer Platforms (IDPs). Were creating the aforementioned Terraform modules with all our infra and code standards, thus bringing all this velocity to our DevOps. The Proton plugin for Backstage provides one of these interface elements, a condensed view of a Proton service associated with the entity being displayed including its version information, deployment status, and service instances. Its use of TypeScript proved to be very helpful when mapping client-side JSON structures to those in server-side Golang. The framework provides many opportunities for developers to create GUIs for Cloud Native applications. Creating a template is just the beginning of the development flow, after that you have CI/CD and second-day operations. I developed a cluster provisioner as a GitHub Action. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page: https://www.linuxfoundation.org/trademark-usage. (** Infragod is our Terraform solution in Wix, you can check my meetup on Infragod here). Build, test, and deploy your code right from GitHub. Plugin includes an Entity ComponentCard, Backend API route and scaffolder actions iLert is a platform for alerting, on-call management and uptime monitoring targeted at DevOps and IT teams. With Proton, customers can standardize centralized templates to meet security, cost, and compliance goals. The Using the AWS Proton plugins for Backstage tutorial, included in the GitHub repository for plugins, provides instructions on how to start using the plugin with your AWS account. All rights reserved. Grade the quality of your Backstage services using Scorecards. And how to design those abstraction layers when building your own IDP? Combining Backstage for service discovery with Humanitec for infrastructure orchestration, deployment and dependency management, teams can achieve a new degree of Ops automation and developer self-service on all levels. The first phase of backstage was a UI framework. Humanitec lets them create baseline configurations and golden paths, so developers can interact independently and effortlessly with the underlying infrastructure. Load users and groups from Okta into the Backstage catalog. Well also look at how top performing engineering organizations combine service catalog functionality with Internal Developer Platforms (IDPs) to provide their engineers with an end-to-end development and deployment experience of the highest quality. A marketplace where engineers can propose projects suitable for inner sourcing. Although some people seem to think they are mutually exclusive, IDPs and service catalogs (or Humanitec and Backstage) actually complement one another quite well. This view is especially interesting for the team owning said service, as it makes it easy to learn about any missing functionalities or potential new features. Service catalogs and Backstage in particular provide enterprise teams with an incredibly useful pane of glass on top of their apps and services. Core features that are good places to get started with Backstage. Engineers can now not only one-click create a new service with all required metadata attached to it, but also one-click deploy it to a new environment, provisioned with the resources they need. Spotify will offer paid plugins for Backstage in 2022. Backstage gives teams a very straightforward method to unify all of your infrastructure tooling, services, and documentation under a single, easy-to-use interface. For example, app developers can create Backstage components that will be registered in the Backstage software catalog through the plugin, powered by AWS Proton service templates. With Proton, platform engineers can associate their CI/CD pipeline, environment, and service templates together while only surfacing required inputs to developers. We look at the limitations of service catalogs and how Internal Developer Platforms complement them to unlock true developer self-service. It allows anyone in the enterprise to find out very quickly whether a certain required service is already available to then coordinate directly with the respective responsible team. From there, you might have a CI pipeline with GitHub Actions and JFrog Artifactory in place. Taras and Charles from the Frontside team demonstrate their solution for testing ingestors in Backstage. Easily view and test your gRPC API with a GUI Client, inspired from BloomRPC application. Its also extremely difficult to map out similar if not identical services to avoid duplication and prevent engineers from reinventing the wheel over and over again across multiple teams. Writing the live code (environments, states etc) that uses those modules. For a list of trademarks of The Linux Foundation, please see the Trademark Usage page. What is Dynamic Configuration Management? However, an application also needs infrastructure it can be deployed towhether its a containerized environment like Kubernetes, Amazon Elastic Container Service, a set of virtual machines, or some other abstraction. One of the best tools out there that can help you standardize the development flow with CI/CD and second-day operation are the developer portals. Head of Production Infrastructure and Apollo Platform at Palantir, Distinguished Engineer for Defence Unicorns. Ops teams also use service catalogs as a way to define templates and blueprints for the rest of the engineering organization to use. Automate your development process with CI hosted in the cloud or on a private server. We define a service catalog as a means of centralizing all services that are important to the stakeholders of an organization that implements and uses it. In the context of global hyperscalers like GCP and AWS, a service catalog represents the sum of all services that are available in the respective consoles a.k.a. Visualize, understand and optimize your team's tech health. Spotify actually said they reduced onboarding time by more than 50% since introducing Backstage internally. Backstage gave me a fresh experience for developing complex front-end applications by letting me focus on my essential tasks. This is specifically useful in the event vulnerabilities are found in a given service version, as teams can be warned and only the affected environments or apps can be shut down/rolled back. The plugin is already merged to Backstage upstream. Missing a plugin for your favorite tool? Should you use CPU limits or not? Identify any active incidents for an entity and the escalation policy. Built around the concept of metadata YAML files, Backstage makes it easy for a single team to manage tens of services and allows a company to easily manage thousands of them. This lets platform engineers use Proton to view deployment versions and push updates. By making services and their metadata easy to understand and reuse throughout the entire organization, service catalogs bring back a level of transparency and observability that most enterprise teams have long dreamed of regaining. Join this webinar with Viktor Farcic, host of the Youtube channel DevOps Toolkit and co-host of DevOps Paradox. Proton amplifies platform engineer productivity, as they can scale to serve a large development team with templates for both environment and service infrastructure. Learn how to build your IDP with Humanitec. A template using this action will provide feedback in the Task Activity screen when a developer creates a component: The following diagram illustrates how the various components within the scaffolding system work together with this action: Composability is a central principle to Backstage. Do you have an existing plugin that you want to add to the Marketplace? You can get going with the coding right away, fantastic! View Allure reports for your components in Backstage. Allows a user to have easy access to pages within a Backstage app by storing them as "shortcuts" in the sidebar. It is common for these platforms to simplify how developers interact with underlying technology, like container orchestrators, and codify organizational standards and best practices in a manner that is easy to consume. Benchmark and improve the accessibility, performance, SEO, and best practices of your website. Provides Stack Overflow specific functionality that can be used in different ways (e.g. What is an Internal Developer Platform and do you need one? We talk to hundreds of engineering teams and organizations of all sizes every month. Videos, Tutorials, Guides and other resources, Explore all features and functionality in detail, Latest & greatest in DevOps and Platforms, See Humanitecs security overview and practices. Visualize a list secrets stored in your vault instance. Most of the time templates like that are a part of something bigger. Manage all your services and software components in one place regardless of where they are running. They can do it via makefile or a script in package.json, it all depends on what they are using. Given its digital implementation, the service catalog acts, at a minimum, as a digital registry and a means for highly distributed enterprises to see, find, invoke, and execute services regardless of where they exist in the company. It drives how Backstage aggregates information from disparate information sources for developers to view in a single location. Depending on the company, additional information may be essential. View Firebase Functions details for your service in Backstage. Crucially, this means that people in one part of the world can find and utilize the same services that people in other teams use on the other side of the world/enterprise, eliminating the need to develop and support local services. Platform engineers no longer need to use home-grown tools to track updates across their deployed services. In big companies when you need to handle lots of scenarios with different teams working separately, each using different programming languages, things often get quite disconnected. The software then creates an automated PR in an existing/new repo using your code standards, thats in addition to all the things that need to be added in order to take your service up to production (CI/CD yamls, K8s manifests, etc). Additional open source plugins built by Spotify to help you create, manage, and explore your software ecosystem. Here you can find all .NET related actions to improve your scaffolder. Lets look at an example. What I needed was a front-end framework that could help get things done quickly without a lot of complexity between the front-end and back-end. Visualize the your company's official guidelines of different areas of software development. Discover valuable insights hiding inside Xcodes build logs. This will illustrate how Backstage and Proton can be integrated to bootstrap a new application that deploys a static Nginx workload to ECS Fargate across multiple environments, including a CI/CD pipeline provided by AWS CodePipeline. Automate your development process with CI hosted in the cloud or on a private server. Developers can self-serve any tech they need, like DBs, ingress, file storage and all other dependencies their apps require to run. See hidden risks and social patterns in your code. for homepage and search) to compose your Backstage App. View GitLab pipelines, merge requests, languages and contributors. Solutions Architect with Amazon Web Services. View Buildkite CI builds for your service in Backstage. Build, test, and deploy your code right from GitHub. It is no wonder then that ever since the open source announcement, Backstage has quickly become the go-to framework for most enterprises looking to build a service catalog. GitOps is a cluster operational model that makes it easy to spin up and manage fully configured Kubernetes platforms and the applications that run on them. How can we reduce the complexity of versioned ecosystem components for developers? A service catalog like Backstage allows you to easily search all your services and immediately create a new one if what you are looking for is not available. This page features open source plugins built by Spotify that you can add to your Backstage deployment. PagerDuty offers a simple way to identify any active incidents for an entity and the escalation policy. Looking for more plugins? Copyright 2022 Backstage Project Authors. The Linux Foundation has registered trademarks and uses trademarks. Not to mention production code, Infrastructure as code, etc. This plugin will show you information about Docker images within the Harbor cloud native registry. I had a very successful trial with Backstage on a previous project where I learned a lot. Backstage not only shines through a rich set of UI components, but it also provides conventions for defining APIs to talk to backend services. Both methods allow users to leverage the underlying architecture in order to develop highly extensible applications. What do you want to talk about? A service catalog answers the question around which service (or fork of it) is consumed by which applications. What such a setup does not allow you to do however, is running your service. After installing the plugin, this view can be activated for any entity by annotating the corresponding YAML like so: This functionality works by calling the AWS Proton API to retrieve data about the Proton service, and renders it with the user interface primitives provided by Backstage. Then platform engineers can simply add the Proton plugin to their Backstage portal, allowing application developers to interact with Proton templates through the same portal they are using to monitor application metrics or search documentation. API Linter is a quality assurance tool that checks the compliance of API's specifications to Zalando's API rules. Sometimes the testing framework will be different, and sometimes the CI and CD tools will change, not to mention shared modules and documentation. Why all the debate about something that seems so simple? Easily view your New Relic Dashboards in Backstage, via real-time snapshots of your dashboards. He covered AWS, Terraform, Flux and Helm. Proton helps platform engineers scale up their impact with a self-service model, resulting in higher velocity for the development and deployment process throughout an application lifecycle. Open source plugins that you can add to your Backstage deployment. Dominik showed us how to use the Backstage API docs plugin to discover APIs and get rendered documentation from OpenAPI, AsyncAPI, and GraphQL schemas.