Scraping og processering af sager fra Domsdatabasen.
Hver enkelt dom kan tilgås via https://domsdatabasen.dk/#sag/<nummer>, hvor <nummer> er mellem 1 og 3821 (pr. 11-10-2023).
Datasættet ligger i en processeret udgave på Huggingface.
Se src/scripts/scrape.py
.
Se src/scripts/process.py
.
Se src/scripts/finalize.py
.
Developers:
pip install -r requirements.txt
to install the required packages.In the src
directory there are two subdirectories, domsdatabasen
and scripts
. This is a brief explanation of the differences between the two.
All Python files in the domsdatabasen
directory are modules
internal to the project package. Examples here could be a general data loading script,
a definition of a model, or a training function. Think of modules as all the building
blocks of a project.
When a module is importing functions/classes from other modules we use the relative import notation - here’s an example:
from .other_module import some_function
Python files in the scripts
folder are scripts, which are short code snippets that
are external to the project package, and which is meant to actually run the code. As
such, only scripts will be called from the terminal. An analogy here is that the
internal numpy
code are all modules, but the Python code you write where you import
some numpy
functions and actually run them, that a script.
When importing module functions/classes when you’re in a script, you do it like you would normally import from any other package:
from domsdatabasen import some_function
Note that this is also how we import functions/classes in tests, since each test Python file is also a Python script, rather than a module.
Run make test
to test your code, which also updates the “coverage badge” in the
README, showing you how much of your code base that is currently being tested.
Github CI pipelines are included in the repo, running all the tests in the tests
directory, as well as building online documentation, if Github Pages has been enabled
for the repository (can be enabled on Github in the repository settings).
Code Spaces is a new feature on Github, that allows you to develop on a project
completely in the cloud, without having to do any local setup at all. This repo comes
included with a configuration file for running code spaces on Github. When hosted on
alexandrainst/domsdatabasen
then simply press the <> Code
button
and add a code space to get started, which will open a VSCode window directly in your
browser.
.
├── .devcontainer
│ └── devcontainer.json
├── .github
│ └── workflows
│ ├── ci.yaml
│ └── docs.yaml
├── .gitignore
├── .name_and_email
├── .pre-commit-config.yaml
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── Dockerfile
├── LICENSE
├── README.md
├── config
│ ├── __init__.py
│ ├── config.yaml
│ └── hydra
│ └── job_logging
│ └── custom.yaml
├── data
├── gfx
│ └── alexandra_logo.png
├── makefile
├── models
├── notebooks
├── poetry.toml
├── pyproject.toml
├── src
│ ├── scripts
│ │ ├── fix_dot_env_file.py
│ │ └── your_script.py
│ └── domsdatabasen
│ ├── __init__.py
│ └── your_module.py
└── tests
├── __init__.py
└── test_dummy.py