Make OpenverseWP Community

Applying ECS to the ingestion server/data refresh

September 2, 2022

This was a passing thought I had that I wanted to note somewhere. Currently the ingestion server is a small Falcon app that runs most aspects of the data refresh, but then also (in staging/prod) interacts with a fleet of “indexer worker” EC2 instances when performing the Postgres -> Elasticsearch indexing.

FacebookXRedditPinterestEmail
This post was originally published on Make Openverse by Madison.
Read Full Post

You may also like

Follow us

Don't be shy, get in touch. We love meeting interesting people and making new friends.

Privacy settings

Decide which cookies you want to allow.

You can change these settings at any time. However, this can result in some functions no longer being available. For information on deleting the cookies, please consult your browser’s help function.

Learn more about the cookies we use.

With the slider, you can enable or disable different types of cookies:

This website will:

  • Remember which cookies group you accepted
  • Essential: Remember your cookie permission setting
  • Essential: Allow session cookies
  • Essential: Gather information you input into a contact forms, newsletter and other forms across all pages
  • Essential: Keep track of what you input in a shopping cart
  • Essential: Authenticate that you are logged into your user account
  • Essential: Remember language version you selected

This website won't:

  • Remember your login details
  • Functionality: Remember social media settings
  • Functionality: Remember selected region and country
  • Analytics: Keep track of your visited pages and interaction taken
  • Analytics: Keep track about your location and region based on your IP number
  • Analytics: Keep track of the time spent on each page
  • Analytics: Increase the data quality of the statistics functions
  • Advertising: Tailor information and advertising to your interests based on e.g. the content you have visited before. (Currently we do not use targeting or targeting cookies.
  • Advertising: Gather personally identifiable information such as name and location

This website will:

  • Remember which cookies group you accepted
  • Essential: Remember your cookie permission setting
  • Essential: Allow session cookies
  • Essential: Gather information you input into a contact forms, newsletter and other forms across all pages
  • Essential: Keep track of what you input in a shopping cart
  • Essential: Authenticate that you are logged into your user account
  • Essential: Remember language version you selected
  • Functionality: Remember social media settings
  • Functionality: Remember selected region and country

This website won't:

  • Remember your login details
  • Analytics: Keep track of your visited pages and interaction taken
  • Analytics: Keep track about your location and region based on your IP number
  • Analytics: Keep track of the time spent on each page
  • Analytics: Increase the data quality of the statistics functions
  • Advertising: Tailor information and advertising to your interests based on e.g. the content you have visited before. (Currently we do not use targeting or targeting cookies.
  • Advertising: Gather personally identifiable information such as name and location

This website will:

  • Remember which cookies group you accepted
  • Essential: Remember your cookie permission setting
  • Essential: Allow session cookies
  • Essential: Gather information you input into a contact forms, newsletter and other forms across all pages
  • Essential: Keep track of what you input in a shopping cart
  • Essential: Authenticate that you are logged into your user account
  • Essential: Remember language version you selected
  • Functionality: Remember social media settings
  • Functionality: Remember selected region and country
  • Analytics: Keep track of your visited pages and interaction taken
  • Analytics: Keep track about your location and region based on your IP number
  • Analytics: Keep track of the time spent on each page
  • Analytics: Increase the data quality of the statistics functions

This website won't:

  • Remember your login details
  • Advertising: Use information for tailored advertising with third parties
  • Advertising: Allow you to connect to social sites
  • Advertising: Identify device you are using
  • Advertising: Gather personally identifiable information such as name and location

This website will:

  • Remember which cookies group you accepted
  • Essential: Remember your cookie permission setting
  • Essential: Allow session cookies
  • Essential: Gather information you input into a contact forms, newsletter and other forms across all pages
  • Essential: Keep track of what you input in a shopping cart
  • Essential: Authenticate that you are logged into your user account
  • Essential: Remember language version you selected
  • Functionality: Remember social media settings
  • Functionality: Remember selected region and country
  • Analytics: Keep track of your visited pages and interaction taken
  • Analytics: Keep track about your location and region based on your IP number
  • Analytics: Keep track of the time spent on each page
  • Analytics: Increase the data quality of the statistics functions
  • Advertising: Use information for tailored advertising with third parties
  • Advertising: Allow you to connect to social sitesl Advertising: Identify device you are using
  • Advertising: Gather personally identifiable information such as name and location

This website won't:

  • Remember your login details
Save & Close