Why is it a pain to inject dependencies manually? I think this is because people assume for some reason that a class isn't allowed to instantiate its own dependencies.
It means that in most cases, you just call a static factory method like create() rather than the constructor. This method will default to using Instance.now() but also gives you a way to provide your own now() for tests (or other non-standard situations).
At the top of the call stack, you call App.create() and boom - you have a whole tree of dependencies.
If the class instantiates its own dependencies then, by definition, you're not injecting those dependencies, so you're not doing dependency injection at all!
That seems too dogmatic. Does the fact that Foo has a static function that knows how to create its dependencies disqualify the class from being a case of DI?
Yes because injecting nowFn is trivial and not a case for DI.
Consider a database handle, or network socket, or http response payload. Clearly each class shouldn't be making its own version of those.
You're nitpicking for no good reason. You can create global handles to each of those items, and let them instantiate with the class or override them with a create function.
Dependency injection boils down the question of whether or not you can dynamically change a dependency at runtime.
many of the things I dependeon are shared services where there should only be one instance. Singleton means a global variable. a di framework lets me have on without a global - meaning if I need a second service I can do it with just a change to how the di works.
there is no right answer of course. Time should be a global so that all timers/clocks advance in lock step. I hav a complex fake time system that allows my tests to advance minutes at a time without waiting on the wall clock. (If you deal with relativity this may not work - for everyone else I encourage it)
Maybe I'm misunderstanding what you're saying but a connection pool seems like almost a canonical example of something that shouldn't be a singleton. You might want connection pools that connect to different databases or to the same database in read-only vs. read/write mode, etc.
I meant "singleton" in the sense of a single value for a type shared by anything that requires one, i.e. a Guice singleton ( https://github.com/google/guice/wiki/scopes#singleton ) not a value in global scope. Or maybe a single value by type with an annotation, the salient point is that there are values in a program that must be shared for correctness. Parameterless constructors prohibit you from using these (unless you have global variables).
Then these different pools can be separate singletons. You still don't want to instantiate multiple identical pools.
You can use the type system to your advantage. Cut a new type and inject a ReadOnlyDataSource or a SecondDatabaseDataSource or whatnot. Figure out what should only have one instance in your app, wrap a type around it, put it in the singleton scope, and inject that.
This has the advantage that you don't need an extra framework/dependency to handle DI, and it means that dependencies are usually much easier to trace (because you've literally got all the code in your project, no metaprogramming or reflection required). There are limits to this style of DI, but in practice I've not reached those limits yet, and I suspect if you do reach those limits, your DI is just too complicated in the first place.
I think most people using these frameworks are aware that DI is just automated instantiation. If your program has a limited number of ways of composing instantiations, it may not be useful to you. The amount of ceremony reduced may not be worth the overhead.
This conversation repeats itself ad infinitum around DI, ORMs, caching, security, logging, validation, etc, etc, etc... no, you don't need a framework. You can write your own. There are three common outcomes of this:
* Your framework gets so complicated that someone rips it out and replaces it with one of the standard frameworks.
* Your framework gets so complicated that it turns into a popular project in its own right.
* Your app dies and your custom framework dies with it.
I'm not suggesting a custom framework here, I'm suggesting no DI framework at all. No reflection, no extra configuration, no nothing, just composing classes manually using the normal structure of the language.
At some point this stops working, I agree — this isn't necessarily an infinitely scalable solution. At that point, switching to (1) is usually a fairly simple endeavour because you're already using DI, you just need to wire it together differently. But I've been surprised at the number of cases where just going without a framework altogether has been completely sufficient, and has been successful for far longer than I'd have originally expected.
If you construct an object once, then what is the difference between that and a singleton?
The answer is scope. Singletons exist explicitly for the purpose of violating scoping for the convenience of not having to thread that dependency through constructors.
Calling these "singletons" maybe created confusion here, I'm more talking about singleton in the sense of a value that is created once and shared (i.e. injecting a value as a "Singleton" via something like Guice). I'm not arguing that you need to have values in global scope, I'm arguing that parameterless constructors prevent you from using shared values (actually _unless_ you have singletons in global scope).
Great talk, there's a lot I can relate to in here.
I find this topic difficult to navigate because of the many trade-offs. One aspect that wasn't mentioned is temporal. A lot of the time, it makes sense to start with a "database-oriented design" (in the pejorative sense), where your types are just whatever shape your data has in Postgres.
However, as time goes on and your understanding of the domain grows, you start to realize the limitations of that approach. At that point, it probably makes sense to introduce a separate domain model and use explicit mapping. But finding that point in time where you want to switch is not trivial.
Should you start with a domain model from the get-go? Maybe, but it's risky because you may end up with domain objects that don't actually do a better job of representing the domain than whatever you have in your SQL tables. It also feels awkward (and is hard to justify in a team) to map back and forth between domain model, sql SELECT row and JSON response body if they're pretty much the same, at least initially.
So it might very well be that, rather than starting with a domain model, the best approach is to refactor your way into it once you have a better feel for the domain. Err on the side of little or no abstraction, but don't hesitate to introduce abstraction when you feel the pain from too much "concretion". Again, it takes judgment so it's hard to teach (which the talk does an admirable job in pointing out).
Pretty naive question, but what differentiates a "domain model" from these more primitive data representations? I see the term thrown around a lot but I've never been able to grok what people actually mean.
By domain model do you mean something like what a scientist would call a theory? A description of your domain in terms of some fundamental concepts, how they relate to each other, their behaviour, etc? Something like a specification?
Which could of course have many possible concrete implementations (and many possible ways to represent it with data). Where I get confused with this is I'm not sure what it means to map data to and from your domain model (it's an actual code entity?), so I'm probably thinking about this wrong.
A quick example can be found with date. You can store it in ISO 8601 string and often it makes more sense as this is a shared spec between systems. But when it comes to actually display it, there's a lot of additional concerns that creep in such as localization and timezones. Then you need to have a data structure that split the components, and some components may be used as keys or parameters for some logic that outputs the final representation, also as a string.
So both the storage and presentation layer are strings, but they differs. So to reconcile both, you need an intermediate layer, which will contains structures that are the domain models, and logic that manipulate them. To jump from one layer to another you map the data, in this example, string to structs then to string.
With MVC and CRUD apps, the layers often have similar models (or the same, especially with dynamic languages) so you don't bother with mapping. But when the use cases becomes more complex, they alter the domain layer and the models within. So then you need to add mapping code. Your storage layers may have many tables (if using sql), but then it's a single struct at the domain layer, which then becomes many models at the presentation layer with duplicate information.
NOTE
That's why a lot of people don't like most ORM libraries. They're great when the models are similar, but when they start to diverge, you always need to resort to raw SQL query, then it becomes a pain to refactor. The good ORM libraries relies on metaprogramming, then they're just weird SQL.
Not really. It's all about the code you need to write. Instead of wrangling the data structures you get from the ORM which is usually similar to maps and array of maps. You have something that makes the domain logic cleaner and clear. Code for mapping data are simple, so you just pay the time price for writing them in exchange for having maintainable use case logic.
Generally the ideal format for one problem is not the same as another. For example, to store a graph in a RDBMS, the ideal format is probably an adjacency list with a recursive query to iterate it. But in my app code, it’s probably easiest as an object-graph just pointing at each other. And in the context of my frontend, I don’t even want to talk about the graph, the user can only really talk about one node’s parent/child relationship at a time.
There’s no one data model ideal for all scenarios — so why not have a different model for each scenario? Then I just need to figure out a way to transform between one model and the next, and whatever logic depending on that idealized data model can now be implemented fairly simply (since that’s the nature of a good data model - the rest of the logic often just falls out).
So the data model you’re using then is localized to the domain/subject in question. You’re just transitioning the data between models as needed. A domain just being an arbitrary context — the persistence layer, or the UI logic, or even specific like I want my model for an accountant to reflect how an accountant UI page would organize it because I only understand 30% of what they’re asking me to do so keeping it “in their terms” makes things much easier to implement blindly. Or perhaps the primary purpose of this particular function is various aggregations for reporting, so I start off by organizing my dataset into a hierarchy that largely aligns with the aggregation groups. Once it’s aligned properly, the aggregation logic itself becomes utterly trivial to express
You could even say that every time you query the database beyond a single table select *, you’re creating a new domain-specific data model. You’re just transforming from the original table representations to a new one.
All domain modeling is specifically choosing a representation that best fits the logic you’re about to write, and then figuring out how to take the model you have and turn it into the model you want. Everything else on the subject is just implementation detail.
> For example, to store a graph in a RDBMS, the ideal format is probably an adjacency list with a recursive query to iterate it
I know this was a minor point, but I think it speaks to the overall topic, so I'll poke at it.
Adjacency lists are perhaps the worst way to store a graph / tree in RDBMS. They may be the easiest to understand, but they have some of the worst performance characteristics, especially if your RDBMS doesn't have Recursive CTEs. This starts to matter at a much lower scale than you might think; several million rows is enough to start showing slowdowns.
This book [0] (Joe Celko's Trees and Hierarchies in SQL For Smarties) shows many other options, though it does lack the closure table approach [1], which is my preferred approach.
And here, we come full circle back to the long-held friction between DBs and applications. You start mentioning triggers, and devs flinch, stating that they don't want logic in the DB. In every case I've ever seen, the replacements they come up with is incredibly convoluted and prone to errors, but hey, it's not in the DB. There is no reason to fear triggers, if and only if you treat them the same way that you'd treat code (because it is): added/modified/removed only via PRs, with careful review and testing.
My understanding is, a database model is one that is fully normalized - design tables to have no redundant/repeated piece of information. You know, the one they teach you when you study relational DBs.
In that model, you can navigate from anywhere to anywhere by following references.
The domain model, at least from a DDD perspective, is different in at least a couple of ways: your domain classes expose business behaviours, and you can hide certain entities as such.
For example, imagine an e-commerce application where you have to represent an order.
In the DB model, you will have the `order` table as well as the `order_line` table, where each row of the latter references a row of the former. In your domain model, instead, you might decide to have a single Order class with order lines only accessed via methods and in the form of strings, or tuples, or whatever - just not with an entity. The Order class hides the existence of the order_line table.
Plus, the Order class will have methods such as `markAsPaid()` etc, also hiding the implementation details of how you persist this type of information - an enum? a boolean? another table referencing rows of `order`? It does not matter to callers.
For me domain model means capturing as much information about the domain you are modeling in the types and data structures you use. Most of the time that ends up meaning use Unions to make illegal states unrepresentable. For example, I have not seen a database native approach to saving union types to databases. In that case using another domain layer becomes mandatory.
To me domain model is an Object-Oriented API through which I can interact with the data in the system. Another way to interact would be direct SQL-calls of course, but then users would need to know about how the data is represented in the database-schema. Whereas with an OOP API, API-methods return instances of several multiple model-classes.
The way the different classes are associated with each other by method calls makes evidednt a kind of "theory" of our system, what kind of objects there are in the system what operations they can perform returning other types of objects as results and so on. So it looks much like a "theory" might in Ecological Biology, m ultiple species interacting with each other.
> Should you start with a domain model from the get-go? Maybe, but it's risky because you may end up with domain objects that don't actually do a better job of representing the domain than whatever you have in your SQL tables.
You absolutely should go with a domain model from the get-go. You can take some shortcuts if absolutely necessary, such as simply using a typealias like `type User = PostgresUser`. But you should definitely NOT use postgres-types inside all the rest of your code - that just asks for a terrible refactoring later.
> It also feels awkward (and is hard to justify in a team) to map back and forth between domain model, sql SELECT row and JSON response body if they're pretty much the same, at least initially.
Absolutely not. This is the most normal thing in the world. And, in fact, they won't be the same anyways. Don't you want to use at least decent calendar/datetime types and speaking names? Don't you want to at least structure things a bit? And you should really really use proper types for IDs.
User(name: string, posts: string[]) is terrible.
User(name: UserName, posts: PostId[]) is acceptable. So you will have to do some kind of mapping even in the vast majority of trivial cases.
After decades of experience, I'm starting to acquire a notion that most of modern web app development is simply the obstinate refusal to put the code where it really belongs: inside the database engine.
Impedance mismatch, ORM, type generators, query parameterisation, async, etc... all stem from treating data as this "external" thing instead of the beating heart of the application.
It terrifies me to say this, but sooner or later someone is going to cook up a JavaScript database engine that also has web capability, along with a native client-side cache component... and then it'll be curtains for traditional databases.
Oh, the performance will be atrocious and grey-bearded wise old men will waggle their fingers in warning, but nobody will care. It'll be simple, consistent, integrated, and productive.
I'm not thinking of "current" DB engines, but an entirely new stack that is end-to-end JavaScript (or WASM).
Something like Java's Akka or .NET Orleans combined with React.
So the "data" would be persisted by stateful Actors, which can run either on the server or in the browser, using the exact same code. Actors running in the browser can persist to localStorage and on the server the actors can persist to blob storage or whatever.
Actors have unique addresses that can be used to activate them. In this system these become standard HTTP URIs so that there is a uniform calling convention.
It sounds interesting, but its still just moving the problem surface elsewhere.
Bringing model definition and usage closer to the storage layer thereby reducing the need for translation and transport might cut down on the need for repeated and variant definitions across layers but it doesn't remove the other issues related to data storage.
There will need to be a system for storage and that will have to deal with transactional state management as well as consistency schemes, having that in an actor layer that is shared with other parts of the system might solve some issues, but it'd need to be really carefully managed so as not to inflict the solutions to those problems on the other parts of the system using the same transport mechanisms.
i dislike javascript with a passion but I'd be down for using a wasm based system that does what you say, my skepticism is usually just me shouting at clouds so it'd be interesting to see a working model.
Disclaimer: I am the developer of SpacetimeDB which is spiritually similar. We absolutely intend to run client side as well. We need to for client side prediction. And eventually we’ll probably do web rendering at some point as well.
First and foremost, what if there is not "the" database? What if you have multiple places that store data? For example, a postgres for ACID stuff, something like Kafka/RabbitMQ or similar to easily communicate with other services (or even yourself) - and sure, you could do that in postgres, but it's a trade-off. Then maybe something like redis/memcache for quick lookups/caching, then maybe elasticsearch for indexed search queries, and so on. And you usually also have some http API.
Sure you can say "I just do all with postgres" and honestly, that's often a good choice.
But it shows that it's not where "code (...) really belongs". Even IF you move a lot of logic into your database engine (and you often should), you most of the time will have another API and there will be a connection. Well, unless you use shared database tables with another application for communication.
All you do is pushing it out further to a later point - and often forcefully so.
> It terrifies me to say this, but sooner or later someone is going to cook up a JavaScript database engine that also has web capability, along with a native client-side cache component... and then it'll be curtains for traditional databases.
Not going to happen. Services like https://spacetimedb.com exist. Also, solutions like Spark (where you send your code to the database(s)) exist. And for certain things, they are great. However, it is a trade-off. There is no one-fits-all solution.
These are different things, and the fact that they're so often conflated is IMO prima facie that they're misused. If you need a queue, you shouldn't be reaching for Kafka, and vice-versa.
> Then maybe something like redis/memcache for quick lookups/caching
With proper RDBMS schema, indexing, and queries, these are _very_ frequently not needed. At FAANG scale, sure, but I think most would be shocked at how performant a properly-tuned RBDMS can be. That's of course the problem; RDBMS are incredibly difficult to run optimally at scale, and so the easier solution is to scale them up and slap a cache in front of them.
> elasticsearch for indexed search queries
Again, probably not needed for most. Both Postgres, MySQL, and SQLite (and I assume others) all have FTS that works quite well. ES is massively complicated to administer.
What you are saying isn't all that wrong. But it's besides the point that I'm making. Because it's not required to use ALL (or even more than one) of the tools or things (such as an http API) that I listed.
The Chinese are warning their citizens about travel to the U.S.
> China issued an alert warning its citizens and students of the potential risk of traveling in the U.S. and attending schools there.
>
> “Recently, due to the deterioration of China-US economic and trade relations and the domestic security situation in the United States, the Ministry of Culture and Tourism reminds Chinese tourists to fully assess the risks of traveling to the United States and be cautious,” the ministry said in an alert.
Many EU countries have done the same in recent weeks. This is less about putting diplomatic pressure, and more about warning people to avoid traveling to the US due to real danger.
The US government has already been rounding people up who haven’t committed a crime. Most notably a student from Columbia college. This is a real danger.
They've also been shipping people off to a prison in El Salvador without due process and, according to them, no way to bring them back if they send the wrong person.
Avoiding travel to the US right now is a perfectly reasonable decision.
I have a trip to China on Saturday and I'm worried about:
a. Chinese retaliation. I don't think they are going to start arresting Americans, but it might get uncomfortable if I don't have my passport in the subway station.
b. American retaliation. My wife has a greencard, this is actually worrying me more than anything, although as far as I have heard so far, we should be OK.
Trip is still on. I'm really interested in how China has changed since I left 8 years ago.
I don't think you have lived experience to back that up. There are definitely people being turned away at customs in the US, and China has targeted nationals of other countries they have disputes with (e.g. during the Huawei daughter incident and putting two Canadians in jail for two years while she got house arrest in her mansion). And anyways, having your passport in a Beijing subway station is just the way things are now (never was needed before, but they random check all the time before I left in 2016).
There have been people turned away at customs for as long as you have been alive. These are only stories because they're effective at manipulating you into believing the narrative du jour.
Yes, and I've always been the "it could never happen to me" person. But now things are volatile and chaotic. That is elevated risk, and you just shouldn't ignore it.
The citizens of other countries should be concerned too. Even short term plans could end badly for some disruptive measure Trump because whatever, as he did basically every week since he got in charge.
Yes, I think it's fair to say that nullables are a form of test doubles. But the approach is meaningfully different from other approaches as it avoids the problems that you get from using mocks in the narrow sense of the word: you're only testing against the behavior of the mocks (not the real code) and you need to keep updating the mocks as the system changes, which adds maintenance cost.
I don't really see how production-code mocks differ from library-based mocks. If you add but a single constructor parameter to your class, you already need to change your production-mock/stub too.
I can say without exaggeration that this article changed the way I write software. Many of the concepts are interesting by themselves (embedded stubs, nullables, sociable tests) but together these patterns form a strong approach for writing fast-running tests.
At first I wasn't sure how well this was going to work, but 30,000 lines later I can see how Shore's approach applies beautifully to real-life projects.
The only thing I'm struggling with is getting the rest of the team to adopt Testing Without Mocks. It takes a bit of effort to learn how to use it correctly.
Honestly I'd probably resist this too. Maybe you could practice on me? Help see the benefit?
I'm seeing three main points in this advice:
1. Don't misuse mocks. (Mocks, fakes, whatever.)
2. Write testable, well-factored code with pluggable dependencies.
3. Don't use mocks at all. Use "nullables" instead.
I'm totally on board with (1) and (2) as good advice in general, no matter how you're testing. But (3) seems like an independent choice. What benefit does (3) specifically deliver that you wouldn't get with (1) and (2) and mocks instead of nullables?
I'm ready to admit I'm missing something, but I don't see it.
The beauty of the nullable approach is that when you're testing the system under test (SUT), you're testing it together with its dependencies. Not just that - you're running the real production code of the dependencies (including transitive dependencies!), just with a off switch toggled.
What the off switch does is really just prevent side effects, like making actual HTTP requests to the outside world or reading bytes from the file system. Using embedded stub makes this a breeze (another revelation for me).
For example, while building the "/v1/add-account" endpoint, you write a test for the request handler function. If you write this test using Shore's approach, you'll exercise the whole tree of dependencies, down to the leaves.
This is what is meant by sociable tests - the SUT isn't isolated from its dependencies. It's the real code all the way down, except that you're calling createNull() instead of create() to instantiate the classes. There are many benefits here, but to me the most important ones are: (1) you don't need to write mocks as separate classes, (2) you can catch errors that arise in the interplay of the various classes or modules and (3) as a result you get a lot more confidence that everything is still working after refactoring.
A sociable microtest is a little like an integration test, but you don't exercise any out-of-process dependencies (perhaps with the exception of the RDBMS) so the tests run in milliseconds rather than seconds.
You commented elsewhere that you're worried about the separation of test code and prod code. Yes, this is a bit of a holy cow that I also had trouble adapting to. It turns out, having test-focused code mixed in with your prod classes is totally fine. I'd perhaps prefer a cleaner separation but honestly it's not a big issue in practice.
I'm all aboard this, in fact it's how I like to write my "unit" tests, but still I'm unsure about this nullable and test code with prod.
What I do is I simply mock as little as possible (leaf nodes such as timers, remote calls, sometimes RDBMS/filesystem, ...). But I'm not sure what embedding the mock in the prod code gains you? I wonder if part of it is down to the language used?
For instance it says "Put the stub in the same file as the rest of your code so it’s easy to remember and update when your code changes.". Maybe it's because they're using a dynamic language? I'm using C++, so any change in interface will fail to compile the mock, making this less important. You could change the behaviour whilst keeping the same interface, but it's very rare in my experience (and definitely should be avoided).
Nullable I could see some values in some cases where you also want to be able to disable a functionality in prod. But in that case you have a function in the interface to turn it off, so you might as well use that. I can see where using Nullable in construction avoids having to expose this method through the whole dependency tree, but at the same time you lose the ability to dynamically turn in on/off.
I think the big benefit comes from hiding implementation details away from tests. Here—let me copy-paste an example I used elsewhere in this thread:
---
With mock injection:
m = new FooMock()
m.call("Bar", 1, 2)
m.call("Baz", 3)
p = new Thing(m)
actual = p.doSomething()
assert(expected == actual)
With a test-specific factory method:
p = Thing::makeTestThing(1, 2, 3) // static method
actual = p.doSomething()
assert(expected == actual)
With the test-specific factory, mocked dependencies are encapsulated by Thing. You can configure the their behaviour by passing different parameters to the factory method, but implementation details like `m.call("Bar", 1, 2)` are hidden. The test doesn't need to know that `doSomething` calls `Foo.Bar`.
I'm not sure whether the author of the article would consider `Thing` to be a nullable if `makeTestThing` uses a conventional, automatically-generated `FooMock` under the hood, but I don't think it really matters. To me, the big benefit comes from `Thing` assuming the responsibility to configure and inject its own mocks.
Thanks, this helped me understand some things. Unfortunately I don't have time to write a real reply right now, but I think I'm more convinced than before. Thanks for taking the time.
Part of testing is risk management. With mocks you run the risk of the behaviour of your mocks being out of sync with the actual systems you’re mocking. By moving the test-time behaviour closer to the runtime behaviour, you reduce that risk somewhat. If you have other ways of managing that risk or you just never see that happening, you’re good.
The value-add for (3) is that "nullables" encapsulate all the fiddly details that mocks (tend to) expose.
Here's an example. With mock injection:
m = new FooMock()
m.call("Bar", 1, 2)
m.call("Baz", 3)
p = new Thing(m)
actual = p.doSomething()
assert(expected == actual)
With a test-specific factory method:
p = Thing::makeTestThing(1, 2, 3) // static method
actual = p.doSomething()
assert(expected == actual)
With the test-specific factory, mocked dependencies are encapsulated by Thing. You can configure the their behaviour by passing different parameters to the factory method, but implementation details like `m.call("Bar", 1, 2)` are hidden. The test doesn't need to know that `doSomething` calls `Foo.Bar`.
I'm not sure whether the author of the article would consider `Thing` to be a nullable if `makeTestThing` uses a conventional, automatically-generated `FooMock` under the hood, but I don't think it really matters. To me, the big benefit comes from `Thing` assuming the responsibility to configure and inject its own mocks.
Wonderful report! So now we know how long it takes to solve all of the problems: 729 hrs.
SICP is hard to work through even if you're just reading but wow, the exercises are on another level! I wonder how it compares to, say, a physics or biology textbook
Ha ha, first time I’ve seen another reference to this! Back in the day, I got such a kick out of his description that I began imitating it myself, calling it my “David Lynch Special” dish.
If you lift that assumption, the problem kind of goes away. James Shore calls this Parameterless Instantiation: https://www.jamesshore.com/v2/projects/nullables/testing-wit...
It means that in most cases, you just call a static factory method like create() rather than the constructor. This method will default to using Instance.now() but also gives you a way to provide your own now() for tests (or other non-standard situations).
At the top of the call stack, you call App.create() and boom - you have a whole tree of dependencies.