Migrated website from python to nodejs and merged HAR docs

[max-ostapenko]

• all the httparchive.org content managed by Astro
• most of the website is served as static, moved dates and metrics to API endpoints
• merged har.fyi content under docs/

There were many duplicated and custom operations, like JS management and resources versioning, that are taken care of by framework anyway.

[tunetheweb]

Heads up that this is gonna take me a while to review!

In meantime can you look at CI and CodeQL errors?

[max-ostapenko]

@tunetheweb totally.

FYI I dropped a lot of PHP legacy pages in this rewrite. I think that was too long ago, but let me know - I can double check the logs.

The security suggestions to be finished today/ tomorrow. And I want to check Lighthouse too.

P. S. It's actually the mess with tech reports motivated me to look deeper here. Category dashboard looks very bad with filters applied. Needs a relational database.

[github-advanced-security]

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

• The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
• Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
• You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

[max-ostapenko]

@tunetheweb closed all TODO's.

[max-ostapenko]

@tunetheweb here are the summarized changes.

Preview deployment: https://httparchive--staging-supndlqf.web.app/
Hosting console: https://console.firebase.google.com/project/httparchive/hosting/sites/httparchive/channels/staging

1. Hosting

• Migrated from App Engine to Firebase Hosting: Converted legacy Python/Flask App Engine setup to a modern, serverless static architecture powered by Astro + Firebase Hosting.
• Removed App Engine Code: Deleted app.yaml, server.js, Python server/ files, Jinja templates/, and express dependency.
• Added Firebase Hosting Configs:
  • .firebaserc: Configured project mapping to httparchive.
  • firebase.json: Configured static serving out of dist/, cleanUrls: true, trailingSlash: false, font caching (/static/fonts/**), security headers (CSP, HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy), and default emulator port 8080.

2. Custom 404 & SEO Parity

• Custom 404 Page: Added src/pages/404.astro matching original App Engine 404 template parity (issue link, nav buttons, magnifying glass illustration).
• Starlight Config: Set disable404Route: true in astro.config.mjs so Astro compiles the custom 404 page directly to dist/404.html.
• Enhanced SEO Timestamps: Updated src/layouts/Base.astro to apply last_updated.json date_published and date_modified metadata across all /docs/* routes.

3. Development Workflow

• npm start: Builds site and starts native Firebase Hosting emulator (npx firebase-tools emulators:start --only hosting).
• npm run stage: Deploys to Firebase Hosting staging preview channel (npx firebase-tools hosting:channel:deploy staging).
• npm run deploy: Deploys to Firebase Hosting live production channel (npx firebase-tools deploy --only hosting).
• Removed Redundant Scripts: Removed tools/scripts/deploy.sh and obsolete preview / testdeploy npm commands.

4. GitHub Actions

• firebase-preview.yml: Added workflow to automatically build and deploy preview channels for all incoming Pull Requests using FirebaseExtended/action-hosting-deploy@v0.
• deploy.yml: Updated workflow to deploy to Firebase Hosting live channel on push to main.
• Automated Test Scripts: Updated tools/scripts/run_and_test_website.sh and run_and_test_website.bat to run tests against the Firebase Hosting emulator (100% test pass rate: 54/54 passed).
• Lighthouse CI: Updated tools/scripts/set_lighthouse_urls.sh diff paths to scan src/pages/ and public/sitemap.xml.

5. Code Generators (tools/generate/)

• Extracted Shared Helpers: Moved map_to_astro_page() to tools/generate/shared.js.
• Removed Dead Code & Dependencies: Uninstalled recursive-readdir and removed dead find_asset_files() function.
• Modernized Node APIs: Replaced recursive-readdir with native fs.promises.readdir({ recursive: true }) in generate_timestamps.js and replaced deprecated .substr() in generate_sitemap.js.
• Cleaned last_updated.json: Pruned legacy /static/ asset entries from config/last_updated.json (reduced lines from 403 to 203), focusing purely on content pages, reports, and documentation.

6. UI

• Subdomain Lens Detection: Added client-side subdomain detection in src/pages/reports/[report_id].astro and src/js/report.js so subdomains (e.g. wordpress.httparchive.org) automatically pre-select their corresponding lens.

7. Route Mapping (Flask vs. Astro)

Route Path	Production main (Flask)	Migrated Version (Astro / Firebase)	Output Artifact
/	server/routes.py: index() -> templates/index.html	src/pages/index.astro	dist/index.html
/about	server/routes.py: about() -> templates/about.html	src/pages/about.astro	dist/about/index.html
/faq	server/routes.py: faq() -> templates/faq.html	src/pages/faq.astro	dist/faq/index.html
/reports	server/routes.py: reports() -> templates/reports.html	src/pages/reports/index.astro	dist/reports/index.html
/reports/[report_id]	server/routes.py: report() -> templates/report/report.html	src/pages/reports/[report_id].astro	dist/reports/[report_id]/index.html
/reports/techreport/landing	server/routes.py: techreportlanding() -> templates/techreport/landing.html	src/pages/reports/techreport/landing.astro	dist/reports/techreport/landing/index.html
/reports/techreport/tech	server/routes.py: techreport() -> templates/techreport/techreport.html	src/pages/reports/techreport/tech.astro	dist/reports/techreport/tech/index.html
/reports/techreport/[page_id]	server/routes.py: techreportlanding()	src/pages/reports/techreport/[page_id].astro	dist/reports/techreport/[page_id]/index.html
/docs/*	N/A	src/content/docs/docs/* (Starlight MDX)	dist/docs/*
/404	server/errors.py -> templates/error/404.html	src/pages/404.astro	dist/404.html
/sitemap.xml	server/routes.py -> templates/sitemap.xml	tools/generate/generate_sitemap.js	public/sitemap.xml
/robots.txt	server/routes.py: static_from_root()	public/robots.txt	dist/robots.txt
/favicon.ico	server/routes.py: favicon()	public/favicon.ico	dist/favicon.ico

---

8. HTTP Headers

Production Talisman headers (server/csp.py) were ported 1:1 into firebase.json for all routes (/**).

9. HAR

Migrated into content/docs/docs.

10. Server logic moved to API

See HTTPArchive/tech-report-apis#128

Note: I think it may be more ergonomic to merge API here, keeping closer to frontend.