Literally 30 seconds after posting this, I found my answer. I guess I had to read 21 topics instead of 20 
Thanks to: Aggregator not Aggregating
I have to agree with the topic author though - this is not just unintuitive, but according to my use case, most topics point to the wrong solution. I mean… technically, they are probably correct, but there is no way for someone to see this without already knowing the solution itself. Just saying.
I had to choose my first module as the source module for the aggregator. And my problem was this:
Source: Multiple bundles to single bundle
Source: How to aggregate multiple bundles?
etc.
Hope this helps someone else!