Observatoire des pratiques d'influence

This repository contains the code that allowed to build the data and content of the Observatoire des pratiques d'influence website.

The project is split into four functional parts, most of which live in this repository:

• Data Extraction Code for extracting Social Network posts: This relies on 2 sub projects:
• Data Extractors (see ./data-extractors/README.md) a long running Python CLI that fetches posts from the three supported social networks. It can read tasks and store results either on the local filesystem or in a shared backend (see below).
• Data Extraction API Backend (see ./opi-api/README.md) a Python FastAPI server that centralises task definitions, acquisition, and result storage, letting extraction jobs be distributed across multiple machines.
• Brand annotation: Identifies brands that paid influencers for specific posts.
• Data analysis and reporting: Metabase is used for dashboards and reports.
• Web site: ./frontend/README.md nextjs public website exposing some Metabase dashboards.

Architecture

For a list of all technical components, general data flow and data model see ./docs/architecture.md.

Running things locally

Running the Web Site and Data extraction Api Backend using docker compose

A docker compose exists that starts:

• The Web site ./frontend/README.md
• The Data Extraction API Backend see ./opi-api/README.md
• A local NocoDB used to store extraction results
• A postgres used for 2 things:
  • The NocoDB storage
  • The data extraction api backend extraction tasks storage

One time manual setup required for Data extraction api backend

• Start the docker compose (see below)
• Connect to Noco UI
• Use UI to define a Noco API token
• Create a new Noco base and copy the base id (or copy the id of the default base)
• Run the Noco tables creation script ./noco_setup/create-noco-tables.sh providing the base id and API Token created before
• Modify docker-compose opi-api service > environnement values to use the correct values for NOCODB_API_TOKEN & NOCODB_BASE_ID

Running docker compose

The build and start of docker-compose is wrapped in a make target: When running make up:

• The website will be available at http://localhost:3000/
• Extraction API Backend doc is available at http://localhost:8000/docs#

Running data extraction

See ./data-extractors/README.md

Running brand annotation

See ./brand-annotation/README.md

Staging Environment

Staging environment current setup:

• NocoDB base is hosted in D4G shared NocoDB
• Metabase runs on coolify deployed using predefined coolify service Metabase
• Data Extraction API Backend runs on coolify and is re-deployed from /opi-api/Dockerfileon each commit to main
• Postgres db for Data Extraction API Backend runs on coolify
• Website runs on coolify and is re-deployed using /frontend/Dockerfile on each commit to main