Sign in Subscribe

From Data to Deployment: A Week-Long MLOps Project Forecasting My Home Internet Usage

From Data to Deployment: A Week-Long MLOps Project Forecasting My Home Internet Usage

You can see all of my notes for this project and much more here

For a while now, I've been searching for that "unicorn" role that perfectly blends my passions: the data-wrangling of SQL, the logic of programming, and the systems-level thinking of Linux, Docker, and APIs. I found that in MLOps, and to accelerate my journey into the field, I decided to dive into a hands-on project. I believe in learning by doing, so I gave myself one week to build a complete MLOps pipeline to forecast my home internet bandwidth consumption. My goal was to go from raw data in a database to a deployed model serving live predictions, documenting the entire journey—the wins, the frustrations, and the "aha!" moments.

This post outlines that week-long sprint, where I used my own historical bandwidth data, ZenML to orchestrate the workflow, XGBoost for modelling, and Docker for deployment.

Day 1: The Initial Environment "Fist Fight"

Every project starts with setting up the environment. I began by cloning my project repo and setting up a standard Python virtual environment on my Debian server. To manage the MLOps workflow, I deployed a ZenML server using Docker, allowing me to connect my local development machine to a persistent, remote ZenML instance.

With the setup complete, I wrote my first ZenML pipeline step: a simple Python function to query years of historical bandwidth data from my PostgreSQL database. The next logical step was data validation. After an initial two-hour battle with Great Expectations, I pivoted to pandera, which I found much less convoluted. I had a working validation step in about 15 minutes.

However, my first attempt at training a model and deploying it in a container yielded predictions that made no sense at all. This was my first major roadblock and a sign that I needed to rethink my approach fundamentally.

Day 2-3: The "Aha!" Moment: Realizing I Was Cheating the Model

After some much-needed sleep, it hit me. I was making a classic, rookie mistake: leaking future information to my model.

At its core, leaking future information... happens when your training data contains information that would not actually be available at the time you would use the model to make a prediction in a real-world scenario.

I was trying to predict the total data transfer rate (t_rate) for a future period, but I was feeding the model features like the upload rate (u_rate), download rate (d_rate), and total data difference (t_diff) for that same period. If I already had that information, I wouldn't need a model in the first place! This mistake led to overly optimistic performance metrics during training but made the model completely useless for actual forecasting.

This was a fantastic learning experience. I threw out the flawed approach and started over with proper time-series feature engineering.

Day 4: A Proper Reboot with Time-Series Feature Engineering

To build a useful forecasting model, I needed to give it historical context without revealing the answer. My first challenge was that the time intervals in my data were inconsistent, ranging from 1 hour to 12 hours. pandas made this surprisingly easy to solve. I upsampled the entire dataset to a consistent hourly frequency, using interpolation to fill in the gaps.

With clean, hourly data, I engineered a new set of features:

  • Lagged Features: The t_rate from the previous hour (t_rate_lag_1h), the previous day (t_rate_lag_24h), etc. This gives the model recent context.
  • Rolling Statistics: The mean, standard deviation, min, and max t_rate over various windows (e.g., 24 hours, 7 days). This captures trends and seasonality.
  • Cyclical Time Features: I encoded features like hour and day_of_week using sine and cosine transformations to help the model understand their cyclical nature (e.g., hour 23 is close to hour 0).

The results were immediate and profound. The new lagged features, especially t_rate_lag_1h, became the most important predictors for the model, which intuitively makes sense.

Day 5-6: Refining the Model and Embracing Better Tooling

With a solid feature engineering step in my pipeline, I focused on improving the model itself. I started with RandomizedSearchCV for hyperparameter tuning but quickly switched to Optuna, a more advanced optimization framework. I integrated it into a custom training step that also used TimeSeriesSplit for cross-validation—a crucial technique for evaluating time-series models without leaking data across folds.

While the new model performed much better, with a Mean Absolute Error of around 7-10 KB/sec, I noticed that its recursive predictions (using its own output to predict the next step) would start oscillating after a few hours. This was due to the model's heavy reliance on very recent lagged features. Acknowledging this limitation, I decided to scope the final product to predict just the next hour.

Day 7: Closing the Loop with Automated Docker Deployment

I had all the pieces: data ingestion, validation, feature engineering, and model training. It was time for the final "Ops" step: automated deployment.

I added a final step to my ZenML pipeline, deploy_model_api, which accomplishes the following:

  1. Packages Artifacts: It takes the trained XGBoost model, the feature scaler, and a list of required features.
  2. Automates Docker Lifecycle: Using the docker Python library, the script automatically stops and removes any old running version of the API container.
  3. Builds & Deploys: It builds a new Docker image containing a FastAPI application and the model artifacts, then deploys it as a new container.
  4. Injects Secrets: It pulls database credentials from the ZenML secrets manager and safely injects them as environment variables into the running container.

The result is a single, executable pipeline that goes from raw data to a live API endpoint (http://192.168.88.5:19000/predict/next-hour) that serves the latest version of the model.

Conclusion and What's Next

This week-long project was an incredibly engaging and effective learning experience. I went from a collection of scripts to a fully automated, end-to-end MLOps pipeline that trains a model and deploys it as a containerized API.

Of course, "finished" is a relative term. The real power of an MLOps pipeline is the ability to iterate and improve. My next steps include:

  • Improving the Model: I plan to explore training a model that predicts u_diff and d_diff separately, trying different architectures like Facebook's Prophet, and engineering a cap_utilization feature to see if my usage changes as I approach my monthly data cap.
  • Automating Retraining: Setting up a nightly schedule to automatically run the pipeline, retraining the model on new data and deploying the updated API.
  • Logging Predictions: Storing the model's predictions over time to compare them against actual usage, allowing me to track model performance and degradation with each new version.

Stay tuned for part 2!