List of tools in usage (or in our roadmap)
Web application
Front-End
- React.JS ^18.2.0
- Typescript ^5.3.3
- Ant Design 5.0
Back-End
- Node.JS ^20.0 : for classical API
- rust - cargo ^1.86.0 : for compute / memory intensive
Package Manager
- Yarn ^4.3.0
API franework
- Metrics management : GraphQL[^1]
- Websocket : for server side notification
- S3 file access (including parquet files) with authentification control. [^2]
- Others API (Actions): REST
[^1]: trend 2023 [^2]: S3 Access
Rapid prototyping
- Streamlit ^(TODO)
System engineering
- Sysml V2 => Implementation to plan in 2026
Machine learning framework
- NVIDIA Driver Version: 575.51.03
- CUDA Version: 12.9
- pytorch^(TODO) => Native support
- tensorflow^(TODO) => To be integrated for industrial performance
- jax^(TODO) => Evaluation needed to see if relevant
Python packaging framework
- python ^3.12
- pip : supported for all our package (to be test in our CI pipeline)
- conda : supported for all our package (to be test in our CI pipeline), and use for several package < 2025
- pipx : for installing command tools
- uv ^0.7.2 : prefered package manager from 2025 (faster even if maybe not to use for all ML packages)
Data Storage technology
Storage protocol
- S3 storage
- with cloud provider offer behind a loadbalancer to provide access control
- TODO : open question, mapping real_user to s3_user to challenge
- with minio for in premise or dev environment
- with cloud provider offer behind a loadbalancer to provide access control
Storage format
- parquet ^2.11.0
- rust / python : https://github.com/delta-io/delta-rs ⁰0.25
- typescript : https://github.com/hyparam/hyparquet ^1.13
- datafusion-python ^46.0.0
Database
- PostgreSQL
- Prometheus
- Redis
Data interface to provide
- Databricks : connector to implement TODO
- Snowflake : connector to implement TODO
Execution environment
Container
- docker : format of container images
- k8s / k3s : operation environment in our cloud
- k3d : local deployment of cluster for developper / test purpose
Batch computation
- Slurm
ML workflow executor
- https://dagster.io/platform instead of airflow ? TODO
Dashboarding
- tensorboard : in workbench, check integration constraint with our metrics
- Visdom : TODO : test to be performed
- debiai : TODO integration with klarity
- other inspiration
- IHM to manipulate arrow format : https://perspective.finos.org/guide/perspec
- Simplify Dashboard https://vega.github.io/falcon/flights/
TODO
- use these version to build reference docker image
- create a script to identify new version available