forked from LiveCarta/LiveCartaMeta
78 lines
1.9 KiB
Markdown
78 lines
1.9 KiB
Markdown
# Book Meta Data Parser
|
||
|
||
Microservice which solves only one issue – parse book meta data from our publishers. Not depends on what format publisher stores this data, the service must grub this information and send an array of data to the main application without any formatting. The main idea is to add components for parsing different formats and have the ability to add publishers just by updating config files.
|
||
|
||
|
||
## Version 1.0
|
||
|
||
Added two components for working with CSV and FTP.
|
||
|
||
|
||
## Tech Stack
|
||
|
||
• Docker
|
||
• Python 3.11
|
||
• MongoDb 6.0.2
|
||
• Dynaconf
|
||
• Pydantic
|
||
• MongoEngine
|
||
|
||
|
||
## Folder structure
|
||
|
||
|
||
• app
|
||
◦ components
|
||
◦ configs
|
||
▪ configs.py – keys and url for connection to our main app and creds for service db
|
||
▪ main.json – main config
|
||
▪ sources.json – list of sources with components that they use
|
||
◦ models
|
||
◦ sources
|
||
▪ file_types
|
||
▪ source_types
|
||
|
||
|
||
## Sources configuration
|
||
|
||
To configure a new source you need to update source config by adding the params below:
|
||
|
||
• source_name
|
||
• source //with neccesary params for component
|
||
• parser_type //with neccesary params for component
|
||
|
||
Example for CSV files from FTP:
|
||
```json
|
||
{
|
||
"sources": {
|
||
"McGrawHill": {
|
||
"source_name": "McGrawHill",
|
||
"source": {
|
||
"type": "ftp",
|
||
"ftp_url": "127.0.0.1",
|
||
"ftp_login": "frp_login",
|
||
"ftp_password": "frp_pass",
|
||
"local_files_path": "/app/files/McGrawHill/",
|
||
"file_regex": "*.csv"
|
||
},
|
||
"parser_type": {
|
||
"format": "csv"
|
||
}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
Each source parser starts by crontab by command
|
||
|
||
`python update.py {source_name}`
|
||
|
||
|
||
To see list of source types use command
|
||
|
||
`python update.py -h`
|
||
|
||
|
||
## Run Updates
|
||
|
||
Copy .env.sample to .env and update settings |