hypothesis vs pytest

Accelerate your workflow with Semaphore – fast, reliable, and scalable for your team's success.

• Talk to a developer
• Start building for free
• System Status
• Semaphore Cloud
• Semaphore On-Premise
• Semaphore Hybrid
• Premium support
• Docker & Kubernetes
• vs GitHub Actions
• vs Travis CI
• vs Bitbucket
• CI/CD Learning Tool
• Write with us
• Get started

Getting Started With Property-Based Testing in Python With Hypothesis and Pytest

This tutorial will be your gentle guide to property-based testing. Property-based testing is a testing philosophy; a way of approaching testing, much like unit testing is a testing philosophy in which we write tests that verify individual components of your code.

By going through this tutorial, you will:

• learn what property-based testing is;
• understand the key benefits of using property-based testing;
• see how to create property-based tests with Hypothesis;
• attempt a small challenge to understand how to write good property-based tests; and
• Explore several situations in which you can use property-based testing with zero overhead.

What is Property-Based Testing?

In the most common types of testing, you write a test by running your code and then checking if the result you got matches the reference result you expected. This is in contrast with property-based testing , where you write tests that check that the results satisfy certain properties . This shift in perspective makes property-based testing (with Hypothesis) a great tool for a variety of scenarios, like fuzzing or testing roundtripping.

In this tutorial, we will be learning about the concepts behind property-based testing, and then we will put those concepts to practice. In order to do that, we will use three tools: Python, pytest, and Hypothesis.

• Python will be the programming language in which we will write both our functions that need testing and our tests.
• pytest will be the testing framework.
• Hypothesis will be the framework that will enable property-based testing.

Both Python and pytest are simple enough that, even if you are not a Python programmer or a pytest user, you should be able to follow along and get benefits from learning about property-based testing.

Setting up your environment to follow along

If you want to follow along with this tutorial and run the snippets of code and the tests yourself – which is highly recommendable – here is how you set up your environment.

Installing Python and pip

Start by making sure you have a recent version of Python installed. Head to the Python downloads page and grab the most recent version for yourself. Then, make sure your Python installation also has pip installed. [ pip ] is the package installer for Python and you can check if you have it on your machine by running the following command:

(This assumes python is the command to run Python on your machine.) If pip is not installed, follow their installation instructions .

Installing pytest and Hypothesis

pytest, the Python testing framework, and Hypothesis, the property-based testing framework, are easy to install after you have pip. All you have to do is run this command:

This tells pip to install pytest and Hypothesis and additionally it tells pip to update to newer versions if any of the packages are already installed.

To make sure pytest has been properly installed, you can run the following command:

The output on your machine may show a different version, depending on the exact version of pytest you have installed.

To ensure Hypothesis has been installed correctly, you have to open your Python REPL by running the following:

and then, within the REPL, type import hypothesis . If Hypothesis was properly installed, it should look like nothing happened. Immediately after, you can check for the version you have installed with hypothesis.__version__ . Thus, your REPL session would look something like this:

Your first property-based test

In this section, we will write our very first property-based test for a small function. This will show how to write basic tests with Hypothesis.

The function to test

Suppose we implemented a function gcd(n, m) that computes the greatest common divisor of two integers. (The greatest common divisor of n and m is the largest integer d that divides evenly into n and m .) What’s more, suppose that our implementation handles positive and negative integers. Here is what this implementation could look like:

If you save that into a file, say gcd.py , and then run it with:

you will enter an interactive REPL with your function already defined. This allows you to play with it a bit:

Now that the function is running and looks about right, we will test it with Hypothesis.

The property test

A property-based test isn’t wildly different from a standard (pytest) test, but there are some key differences. For example, instead of writing inputs to the function gcd , we let Hypothesis generate arbitrary inputs. Then, instead of hardcoding the expected outputs, we write assertions that ensure that the solution satisfies the properties that it should satisfy.

Thus, to write a property-based test, you need to determine the properties that your answer should satisfy.

Thankfully for us, we already know the properties that the result of gcd must satisfy:

“[…] the greatest common divisor (GCD) of two or more integers […] is the largest positive integer that divides each of the integers.”

So, from that Wikipedia quote, we know that if d is the result of gcd(n, m) , then:

• d is positive;
• d divides n ;
• d divides m ; and
• no other number larger than d divides both n and m .

To turn these properties into a test, we start by writing the signature of a test_ function that accepts the same inputs as the function gcd :

(The prefix test_ is not significant for Hypothesis. We are using Hypothesis with pytest and pytest looks for functions that start with test_ , so that is why our function is called test_gcd .)

The arguments n and m , which are also the arguments of gcd , will be filled in by Hypothesis. For now, we will just assume that they are available.

If n and m are arguments that are available and for which we want to test the function gcd , we have to start by calling gcd with n and m and then saving the result. It is after calling gcd with the supplied arguments and getting the answer that we get to test the answer against the four properties listed above.

Taking the four properties into account, our test function could look like this:

Go ahead and put this test function next to the function gcd in the file gcd.py . Typically, tests live in a different file from the code being tested but this is such a small example that we can have everything in the same file.

Plugging in Hypothesis

We have written the test function but we still haven’t used Hypothesis to power the test. Let’s go ahead and use Hypothesis’ magic to generate a bunch of arguments n and m for our function gcd. In order to do that, we need to figure out what are all the legal inputs that our function gcd should handle.

For our function gcd , the valid inputs are all integers, so we need to tell Hypothesis to generate integers and feed them into test_gcd . To do that, we need to import a couple of things:

given is what we will use to tell Hypothesis that a test function needs to be given data. The submodule strategies is the module that contains lots of tools that know how to generate data.

With these two imports, we can annotate our test:

You can read the decorator @given(st.integers(), st.integers()) as “the test function needs to be given one integer, and then another integer”. To run the test, you can just use pytest :

(Note: depending on your operating system and the way you have things configured, pytest may not end up in your path, and the command pytest gcd.py may not work. If that is the case for you, you can use the command python -m pytest gcd.py instead.)

As soon as you do so, Hypothesis will scream an error message at you, saying that you got a ZeroDivisionError . Let us try to understand what Hypothesis is telling us by looking at the bottom of the output of running the tests:

This shows that the tests failed with a ZeroDivisionError , and the line that reads “Falsifying example: …” contains information about the test case that blew our test up. In our case, this was n = 0 and m = 0 . So, Hypothesis is telling us that when the arguments are both zero, our function fails because it raises a ZeroDivisionError .

The problem lies in the usage of the modulo operator % , which does not accept a right argument of zero. The right argument of % is zero if n is zero, in which case the result should be m . Adding an if statement is a possible fix for this:

However, Hypothesis still won’t be happy. If you run your test again, with pytest gcd.py , you get this output:

This time, the issue is with the very first property that should be satisfied. We can know this because Hypothesis tells us which assertion failed while also telling us which arguments led to that failure. In fact, if we look further up the output, this is what we see:

This time, the issue isn’t really our fault. The greatest common divisor is not defined when both arguments are zero, so it is ok for our function to not know how to handle this case. Thankfully, Hypothesis lets us customise the strategies used to generate arguments. In particular, we can say that we only want to generate integers between a minimum and a maximum value.

The code below changes the test so that it only runs with integers between 1 and 100 for the first argument ( n ) and between -500 and 500 for the second argument ( m ):

That is it! This was your very first property-based test.

Why bother with Property-Based Testing?

To write good property-based tests you need to analyse your problem carefully to be able to write down all the properties that are relevant. This may look quite cumbersome. However, using a tool like Hypothesis has very practical benefits:

• Hypothesis can generate dozens or hundreds of tests for you, while you would typically only write a couple of them;
• tests you write by hand will typically only cover the edge cases you have already thought of, whereas Hypothesis will not have that bias; and
• thinking about your solution to figure out its properties can give you deeper insights into the problem, leading to even better solutions.

These are just some of the advantages of using property-based testing.

Using Hypothesis for free

There are some scenarios in which you can use property-based testing essentially for free (that is, without needing to spend your precious brain power), because you don’t even need to think about properties. Let’s look at two such scenarios.

Testing Roundtripping

Hypothesis is a great tool to test roundtripping. For example, the built-in functions int and str in Python should roundtrip. That is, if x is an integer, then int(str(x)) should still be x . In other words, converting x to a string and then to an integer again should not change its value.

We can write a simple property-based test for this, leveraging the fact that Hypothesis generates dozens of tests for us. Save this in a Python file:

Now, run this file with pytest. Your test should pass!

Did you notice that, in our gcd example above, the very first time we ran Hypothesis we got a ZeroDivisionError ? The test failed, not because of an assert, but simply because our function crashed.

Hypothesis can be used for tests like this. You do not need to write a single property because you are just using Hypothesis to see if your function can deal with different inputs. Of course, even a buggy function can pass a fuzzing test like this, but this helps catch some types of bugs in your code.

Comparing against a gold standard

Sometimes, you want to test a function f that computes something that could be computed by some other function f_alternative . You know this other function is correct (that is why you call it a “gold standard”), but you cannot use it in production because it is very slow, or it consumes a lot of resources, or for some other combination of reasons.

Provided it is ok to use the function f_alternative in a testing environment, a suitable test would be something like the following:

When possible, this type of test is very powerful because it directly tests if your solution is correct for a series of different arguments.

For example, if you refactored an old piece of code, perhaps to simplify its logic or to make it more performant, Hypothesis will give you confidence that your new function will work as it should.

The importance of property completeness

In this section you will learn about the importance of being thorough when listing the properties that are relevant. To illustrate the point, we will reason about property-based tests for a function called my_sort , which is your implementation of a sorting function that accepts lists of integers.

The results are sorted

When thinking about the properties that the result of my_sort satisfies, you come up with the obvious thing: the result of my_sort must be sorted.

So, you set out to assert this property is satisfied:

Now, the only thing missing is the appropriate strategy to generate lists of integers. Thankfully, Hypothesis knows a strategy to generate lists, which is called lists . All you need to do is give it a strategy that generates the elements of the list.

Now that the test has been written, here is a challenge. Copy this code into a file called my_sort.py . Between the import and the test, define a function my_sort that is wrong (that is, write a function that does not sort lists of integers) and yet passes the test if you run it with pytest my_sort.py . (Keep reading when you are ready for spoilers.)

Notice that the only property that we are testing is “all elements of the result are sorted”, so we can return whatever result we want , as long as it is sorted. Here is my fake implementation of my_sort :

This passes our property test and yet is clearly wrong because we always return an empty list. So, are we missing a property? Perhaps.

The lengths are the same

We can try to add another obvious property, which is that the input and the output should have the same length, obviously. This means that our test becomes:

Now that the test has been improved, here is a challenge. Write a new version of my_sort that passes this test and is still wrong. (Keep reading when you are ready for spoilers.)

Notice that we are only testing for the length of the result and whether or not its elements are sorted, but we don’t test which elements are contained in the result. Thus, this fake implementation of my_sort would work:

Use the right numbers

To fix this, we can add the obvious property that the result should only contain numbers from the original list. With sets, this is easy to test:

Now that our test has been improved, I have yet another challenge. Can you write a fake version of my_sort that passes this test? (Keep reading when you are ready for spoilers).

Here is a fake version of my_sort that passes the test above:

The issue here is that we were not precise enough with our new property. In fact, set(result) <= set(int_list) ensures that we only use numbers that were available in the original list, but it doesn’t ensure that we use all of them. What is more, we can’t fix it by simply replacing the <= with == . Can you see why?I will give you a hint. If you just replace the <= with a == , so that the test becomes:

then you can write this passing version of my_sort that is still wrong:

This version is wrong because it reuses the largest element of the original list without respecting the number of times each integer should be used. For example, for the input list [1, 1, 2, 2, 3, 3] the result should be unchanged, whereas this version of my_sort returns [1, 2, 3, 3, 3, 3] .

The final test

A test that is correct and complete would have to take into account how many times each number appears in the original list, which is something the built-in set is not prepared to do. Instead, one could use the collections.Counter from the standard library:

So, at this point, your test function test_my_sort is complete. At this point, it is no longer possible to fool the test! That is, the only way the test will pass is if my_sort is a real sorting function.

Use properties and specific examples

This section showed that the properties that you test should be well thought-through and you should strive to come up with a set of properties that are as specific as possible. When in doubt, it is better to have properties that may look redundant over having too few.

Another strategy that you can follow to help mitigate the danger of having come up with an insufficient set of properties is to mix property-based testing with other forms of testing, which is perfectly reasonable.

For example, on top of having the property-based test test_my_sort , you could add the following test:

This article covered two examples of functions to which we added property-based tests. We only covered the basics of using Hypothesis to run property-based tests but, more importantly, we covered the fundamental concepts that enable a developer to reason about and write complete property-based tests.

Property-based testing isn’t a one-size-fits-all solution that means you will never have to write any other type of test, but it does have characteristics that you should take advantage of whenever possible. In particular, we saw that property-based testing with Hypothesis was beneficial in that:

This article also went over a couple of common gotchas when writing property-based tests and listed scenarios in which property-based testing can be used with no overhead.

If you are interested in learning more about Hypothesis and property-based testing, we recommend you take a look at the Hypothesis docs and, in particular, to the page “What you can generate and how” .

CI/CD Weekly Newsletter 🔔

Semaphore Uncut Podcast 🎙️

Learn ci/cd.

Level up your developer skills to use CI/CD at its max.

5 thoughts on “ Getting Started With Property-Based Testing in Python With Hypothesis and Pytest ”

Awesome intro to property based testing for Python. Thank you, Dan and Rodrigo!

Greeting! Unfortunately, I don’t understand due to translation difficulties. PyCharm writes error messages and does not run the codes. The installation was done fine, check ok. I created a virtual environment. I would like a single good, usable, complete code, an example of what to write in gcd.py and what in test_gcd.py, which the development environment runs without errors. Thanks!

Thanks for article!

“it is better to have properties that may look redundant over having too few” Isn’t it the case with: assert len(result) == len(int_list) and: assert Counter(result) == Counter(int_list) ? I mean: is it possible to satisfy the second condition without satisfying the first ?

Yes. One case could be if result = [0,1], int_list = [0,1,1], and the implementation of Counter returns unique count.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

CI/CD Weekly Newsletter

🔔 Get notified when the new articles and interviews are out.

Test faster, fix more

How do I use pytest fixtures with Hypothesis?

pytest is a great test runner, and is the one Hypothesis itself uses for testing (though Hypothesis works fine with other test runners too).

It has a fairly elaborate fixture system , and people are often unsure how that interacts with Hypothesis. In this article we’ll go over the details of how to use the two together.

Mostly, Hypothesis and py.test fixtures don’t interact: Each just ignores the other’s presence.

When using a @given decorator, any arguments that are not provided in the @given will be left visible in the final function:

This then outputs the following:

We’ve hidden the arguments ‘a’ and ‘c’, but the unspecified arguments ‘b’ and ‘d’ are still left to be passed in. In particular, they can be provided as py.test fixtures:

This also works if we want to use @given with positional arguments:

The positional argument fills in from the right, replacing the ‘a’ argument and leaving us with ‘stuff’ to be provided by the fixture.

Personally I don’t usually do this because I find it gets a bit confusing - if I’m going to use fixtures then I always use the named variant of given. There’s no reason you can’t do it this way if you prefer though.

@given also works fine in combination with parametrized tests:

This will run 3 tests, one for each value for ‘stuff’.

There is one unfortunate feature of how this interaction works though: In pytest you can declare fixtures which do set up and tear down per function. These will “work” with Hypothesis, but they will run once for the entire test function rather than once for each time given calls your test function. So the following will fail:

The counter will not get reset at the beginning of each call to the test function, so it will be incremented each time and the test will start failing after the first call.

There currently aren’t any great ways around this unfortunately. The best you can really do is do manual setup and teardown yourself in your tests using Hypothesis (e.g. by implementing a version of your fixture as a context manager).

Long-term, I’d like to resolve this by providing a mechanism for allowing fixtures to be run for each example (it’s probably not correct to have every function scoped fixture run for each example), but for now it’s stalled because it requires changes on the py.test side as well as the Hypothesis side and we haven’t quite managed to find the time and place to collaborate on figuring out how to fix this yet.

Table of Contents

Testing your python code with hypothesis, installing & using hypothesis, a quick example, understanding hypothesis, using hypothesis strategies, filtering and mapping strategies, composing strategies, constraints & satisfiability, writing reusable strategies with functions.

• @composite: Declarative Strategies
• @example: Explicitly Testing Certain Values

Hypothesis Example: Roman Numeral Converter

I can think of a several Python packages that greatly improved the quality of the software I write. Two of them are pytest and hypothesis . The former adds an ergonomic framework for writing tests and fixtures and a feature-rich test runner. The latter adds property-based testing that can ferret out all but the most stubborn bugs using clever algorithms, and that’s the package we’ll explore in this course.

In an ordinary test you interface with the code you want to test by generating one or more inputs to test against, and then you validate that it returns the right answer. But that, then, raises a tantalizing question: what about all the inputs you didn’t test? Your code coverage tool may well report 100% test coverage, but that does not, ipso facto , mean the code is bug-free.

One of the defining features of Hypothesis is its ability to generate test cases automatically in a manner that is:

Repeated invocations of your tests result in reproducible outcomes, even though Hypothesis does use randomness to generate the data.

You are given a detailed answer that explains how your test failed and why it failed. Hypothesis makes it clear how you, the human, can reproduce the invariant that caused your test to fail.

You can refine its strategies and tell it where or what it should or should not search for. At no point are you compelled to modify your code to suit the whims of Hypothesis if it generates nonsensical data.

So let’s look at how Hypothesis can help you discover errors in your code.

You can install hypothesis by typing pip install hypothesis . It has few dependencies of its own, and should install and run everywhere.

Hypothesis plugs into pytest and unittest by default, so you don’t have to do anything to make it work with it. In addition, Hypothesis comes with a CLI tool you can invoke with hypothesis . But more on that in a bit.

I will use pytest throughout to demonstrate Hypothesis, but it works equally well with the builtin unittest module.

Before I delve into the details of Hypothesis, let’s start with a simple example: a naive CSV writer and reader. A topic that seems simple enough: how hard is it to separate fields of data with a comma and then read it back in later?

But of course CSV is frighteningly hard to get right. The US and UK use '.' as a decimal separator, but in large parts of the world they use ',' which of course results in immediate failure. So then you start quoting things, and now you need a state machine that can distinguish quoted from unquoted; and what about nested quotes, etc.

The naive CSV reader and writer is an excellent stand-in for any number of complex projects where the requirements outwardly seem simple but there lurks a large number of edge cases that you must take into account.

Here the writer simply string quotes each field before joining them together with ',' . The reader does the opposite: it assumes each field is quoted after it is split by the comma.

A naive roundtrip pytest proves the code “works”:

And evidently so:

And for a lot of code that’s where the testing would begin and end. A couple of lines of code to test a couple of functions that outwardly behave in a manner that anybody can read and understand. Now let’s look at what a Hypothesis test would look like, and what happens when we run it:

At first blush there’s nothing here that you couldn’t divine the intent of, even if you don’t know Hypothesis. I’m asking for the argument fields to have a list ranging from one element of generated text up to ten. Aside from that, the test operates in exactly the same manner as before.

Now watch what happens when I run the test:

Hypothesis quickly found an example that broke our code. As it turns out, a list of [','] breaks our code. We get two fields back after round-tripping the code through our CSV writer and reader — uncovering our first bug.

In a nutshell, this is what Hypothesis does. But let’s look at it in detail.

Simply put, Hypothesis generates data using a number of configurable strategies . Strategies range from simple to complex. A simple strategy may generate bools; another integers. You can combine strategies to make larger ones, such as lists or dicts that match certain patterns or structures you want to test. You can clamp their outputs based on certain constraints, like only positive integers or strings of a certain length. You can also write your own strategies if you have particularly complex requirements.

Strategies are the gateway to property-based testing and are a fundamental part of how Hypothesis works. You can find a detailed list of all the strategies in the Strategies reference of their documentation or in the hypothesis.strategies module.

The best way to get a feel for what each strategy does in practice is to import them from the hypothesis.strategies module and call the example() method on an instance:

You may have noticed that the floats example included inf in the list. By default, all strategies will – where feasible – attempt to test all legal (but possibly obscure) forms of values you can generate of that type. That is particularly important as corner cases like inf or NaN are legal floating-point values but, I imagine, not something you’d ordinarily test against yourself.

And that’s one pillar of how Hypothesis tries to find bugs in your code: by testing edge cases that you would likely miss yourself. If you ask it for a text() strategy you’re as likely to be given Western characters as you are a mishmash of unicode and escape-encoded garbage. Understanding why Hypothesis generates the examples it does is a useful way to think about how your code may interact data it has no control over.

Now if it were simply generating text or numbers from an inexhaustible source of numbers or strings, it wouldn’t catch as many errors as it actually does . The reason for that is that each test you write is subjected to a battery of examples drawn from the strategies you’ve designed. If a test case fails, it’s put aside and tested again but with a reduced subset of inputs, if possible. In Hypothesis it’s known as shrinking the search space to try and find the smallest possible result that will cause your code to fail. So instead of a 10,000-length string, if it can find one that’s only 3 or 4, it will try to show that to you instead.

You can tell Hypothesis to filter or map the examples it draws to further reduce them if the strategy does not meet your requirements:

Here I ask for integers where the number is greater than 0 and is evenly divisible by 8. Hypothesis will then attempt to generate examples that meets the constraints you have imposed on it.

You can also map , which works in much the same way as filter. Here I’m asking for lowercase ASCII and then uppercasing them:

Having said that, using either when you don’t have to (I could have asked for uppercase ASCII characters to begin with) is likely to result in slower strategies.

A third option, flatmap , lets you build strategies from strategies; but that deserves closer scrutiny, so I’ll talk about it later.

You can tell Hypothesis to pick one of a number of strategies by composing strategies with | or st.one_of() :

An essential feature when you have to draw from multiple sources of examples for a single data point.

When you ask Hypothesis to draw an example it takes into account the constraints you may have imposed on it: only positive integers; only lists of numbers that add up to exactly 100; any filter() calls you may have applied; and so on. Those are constraints. You’re taking something that was once unbounded (with respect to the strategy you’re drawing an example from, that is) and introducing additional limitations that constrain the possible range of values it can give you.

But consider what happens if I pass filters that will yield nothing at all:

At some point Hypothesis will give up and declare it cannot find anything that satisfies that strategy and its constraints.

Hypothesis gives up after a while if it’s not able to draw an example. Usually that indicates an invariant in the constraints you’ve placed that makes it hard or impossible to draw examples from. In the example above, I asked for numbers that are simultaneously below zero and greater than zero, which is an impossible request.

As you can see, the strategies are simple functions, and they behave as such. You can therefore refactor each strategy into reusable patterns:

The benefit of this approach is that if you discover edge cases that Hypothesis does not account for, you can update the pattern in one place and observe its effects on your code. It’s functional and composable.

One caveat of this approach is that you cannot draw examples and expect Hypothesis to behave correctly. So I don’t recommend you call example() on a strategy only to pass it into another strategy.

For that, you want the @composite decorator.

@composite : Declarative Strategies

If the previous approach is unabashedly functional in nature, this approach is imperative.

The @composite decorator lets you write imperative Python code instead. If you cannot easily structure your strategy with the built-in ones, or if you require more granular control over the values it emits, you should consider the @composite strategy.

Instead of returning a compound strategy object like you would above, you instead draw examples using a special function you’re given access to in the decorated function.

This example draws two randomized names and returns them as a tuple:

Note that the @composite decorator passes in a special draw callable that you must use to draw samples from. You cannot – well, you can , but you shouldn’t – use the example() method on the strategy object you get back. Doing so will break Hypothesis’s ability to synthesize test cases properly.

Because the code is imperative you’re free to modify the drawn examples to your liking. But what if you’re given an example you don’t like or one that breaks a known invariant you don’t wish to test for? For that you can use the assume() function to state the assumptions that Hypothesis must meet if you try to draw an example from generate_full_name .

Let’s say that first_name and last_name must not be equal:

Like the assert statement in Python, the assume() function teaches Hypothesis what is, and is not, a valid example. You use this to great effect to generate complex compound strategies.

I recommend you observe the following rules of thumb if you write imperative strategies with @composite :

If you want to draw a succession of examples to initialize, say, a list or a custom object with values that meet certain criteria you should use filter , where possible, and assume to teach Hypothesis why the value(s) you drew and subsequently discarded weren’t any good.

The example above uses assume() to teach Hypothesis that first_name and last_name must not be equal.

If you can put your functional strategies in separate functions, you should. It encourages code re-use and if your strategies are failing (or not generating the sort of examples you’d expect) you can inspect each strategy in turn. Large nested strategies are harder to untangle and harder still to reason about.

If you can express your requirements with filter and map or the builtin constraints (like min_size or max_size ), you should. Imperative strategies that use assume may take more time to converge on a valid example.

@example : Explicitly Testing Certain Values

Occasionally you’ll come across a handful of cases that either fails or used to fail, and you want to ensure that Hypothesis does not forget to test them, or to indicate to yourself or your fellow developers that certain values are known to cause issues and should be tested explicitly.

The @example decorator does just that:

You can add as many as you like.

Let’s say I wanted to write a simple converter to and from Roman numerals.

Here I’m collecting Roman numerals into numerals , one at a time, by looping over SYMBOLS of valid numerals, subtracting the value of the symbol from number , until the while loop’s condition ( number >= 1 ) is False .

The test is also simple and serves as a smoke test. I generate a random integer and convert it to Roman numerals with to_roman . When it’s all said and done I turn the string of numerals into a set and check that all members of the set are legal Roman numerals.

Now if I run pytest on it seems to hang . But thanks to Hypothesis’s debug mode I can inspect why:

Ah. Instead of testing with tiny numbers like a human would ordinarily do, it used a fantastically large one… which is altogether slow.

OK, so there’s at least one gotcha; it’s not really a bug , but it’s something to think about: limiting the maximum value. I’m only going to limit the test, but it would be reasonable to limit it in the code also.

Changing the max_value to something sensible, like st.integers(max_value=5000) and the test now fails with another error:

It seems our code’s not able to handle the number 0! Which… is correct. The Romans didn’t really use the number zero as we would today; that invention came later, so they had a bunch of workarounds to deal with the absence of something. But that’s neither here nor there in our example. Let’s instead set min_value=1 also, as there is no support for negative numbers either:

OK… not bad. We’ve proven that given a random assortment of numbers between our defined range of values that, indeed, we get something resembling Roman numerals.

One of the hardest things about Hypothesis is framing questions to your testable code in a way that tests its properties but without you, the developer, knowing the answer (necessarily) beforehand. So one simple way to test that there’s at least something semi-coherent coming out of our to_roman function is to check that it can generate the very numerals we defined in SYMBOLS from before:

Here I’m sampling from a tuple of the SYMBOLS from earlier. The sampling algorithm’ll decide what values it wants to give us, all we care about is that we are given examples like ("I", 1) or ("V", 5) to compare against.

So let’s run pytest again:

Oops. The Roman numeral V is equal to 5 and yet we get five IIIII ? A closer examination reveals that, indeed, the code only yields sequences of I equal to the number we pass it. There’s a logic error in our code.

In the example above I loop over the elements in the SYMBOLS dictionary but as it’s ordered the first element is always I . And as the smallest representable value is 1, we end up with just that answer. It’s technically correct as you can count with just I but it’s not very useful.

Fixing it is easy though:

Rerunning the test yields a pass. Now we know that, at the very least, our to_roman function is capable of mapping numbers that are equal to any symbol in SYMBOLS .

Now the litmus test is taking the numeral we’re given and making sense of it. So let’s write a function that converts a Roman numeral back into decimal:

Like to_roman we walk through each character, get the numeral’s numeric value, and add it to the running total. The test is a simple roundtrip test as to_roman has an inverse function from_roman (and vice versa) such that :

Invertible functions are easier to test because you can compare the output of one against the input of another and check if it yields the original value. But not every function has an inverse, though.

Running the test yields a pass:

So now we’re in a pretty good place. But there’s a slight oversight in our Roman numeral converters, though: they don’t respect the subtraction rule for some of the numerals. For instance VI is 6; but IV is 4. The value XI is 11; and IX is 9. Only some (sigh) numerals exhibit this property.

So let’s write another test. This time it’ll fail as we’ve yet to write the modified code. Luckily we know the subtractive numerals we must accommodate:

Pretty simple test. Check that certain numerals yield the value, and that the values yield the right numeral.

With an extensive test suite we should feel fairly confident making changes to the code. If we break something, one of our preexisting tests will fail.

The rules around which numerals are subtractive is rather subjective. The SUBTRACTIVE_SYMBOLS dictionary holds the most common ones. So all we need to do is read ahead of the numerals list to see if there exists a two-digit numeral that has a prescribed value and then we use that instead of the usual value.

The to_roman change is simple. A union of the two numeral symbol dictionaries is all it takes . The code already understands how to turn numbers into numerals — we just added a few more.

This method requires Python 3.9 or later. Read how to merge dictionaries

If done right, running the tests should yield a pass:

And that’s it. We now have useful tests and a functional Roman numeral converter that converts to and from with ease. But one thing we didn’t do is create a strategy that generates Roman numerals using st.text() . A custom composite strategy to generate both valid and invalid Roman numerals to test the ruggedness of our converter is left as an exercise to you.

In the next part of this course we’ll look at more advanced testing strategies.

Unlike a tool like faker that generates realistic-looking test data for fixtures or demos, Hypothesis is a property-based tester . It uses heuristics and clever algorithms to find inputs that break your code.

Testing a function that does not have an inverse to compare the result against – like our Roman numeral converter that works both ways – you often have to approach your code as though it were a black box where you relinquish control of the inputs and outputs. That is harder, but makes for less brittle code.

It’s perfectly fine to mix and match tests. Hypothesis is useful for flushing out invariants you would never think of. Combine it with known inputs and outputs to jump start your testing for the first 80%, and augment it with Hypothesis to catch the remaining 20%.

Be Inspired Sign up and we’ll tell you about new articles and courses

Absolutely no spam. We promise!

Liked the Article?

Why not follow us …, be inspired get python tips sent to your inbox.

We'll tell you about the latest courses and articles.

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications You must be signed in to change notification settings

Hypothesis is a powerful, flexible, and easy to use library for property-based testing.

HypothesisWorks/hypothesis

Folders and files, repository files navigation.

Hypothesis is a family of testing libraries which let you write tests parametrized by a source of examples. A Hypothesis implementation then generates simple and comprehensible examples that make your tests fail. This simplifies writing your tests and makes them more powerful at the same time, by letting software automate the boring bits and do them to a higher standard than a human would, freeing you to focus on the higher level test logic.

This sort of testing is often called "property-based testing", and the most widely known implementation of the concept is the Haskell library QuickCheck , but Hypothesis differs significantly from QuickCheck and is designed to fit idiomatically and easily into existing styles of testing that you are used to, with absolutely no familiarity with Haskell or functional programming needed.

Hypothesis for Python is the original implementation, and the only one that is currently fully production ready and actively maintained.

Hypothesis for Other Languages

The core ideas of Hypothesis are language agnostic and in principle it is suitable for any language. We are interested in developing and supporting implementations for a wide variety of languages, but currently lack the resources to do so, so our porting efforts are mostly prototypes.

The two prototype implementations of Hypothesis for other languages are:

• Hypothesis for Ruby is a reasonable start on a port of Hypothesis to Ruby.
• Hypothesis for Java is a prototype written some time ago. It's far from feature complete and is not under active development, but was intended to prove the viability of the concept.

Additionally there is a port of the core engine of Hypothesis, Conjecture, to Rust. It is not feature complete but in the long run we are hoping to move much of the existing functionality to Rust and rebuild Hypothesis for Python on top of it, greatly lowering the porting effort to other languages.

Any or all of these could be turned into full fledged implementations with relatively little effort (no more than a few months of full time work), but as well as the initial work this would require someone prepared to provide or fund ongoing maintenance efforts for them in order to be viable.

Releases 671

Used by 27.8k.

Contributors 305

• Python 90.2%
• Jupyter Notebook 5.0%

pytest VS hypothesis

Compare pytest vs hypothesis and see what are their differences..

• Local Variables as Accidental Breadcrumbs for Faster Debugging 1 project | news.ycombinator.com | 10 Oct 2024

In this blog post I want to demonstrate how my lab equipment such as a lab power supply or a digital multimeter (DMM) have been integrated into some pytest-based tests. Would love to get your feedback and thoughts! 🚀

Pytest: is a third-party testing framework that supports fixtures, parameterized testing, and easy test discovery while having room to add plugins to extend its functionality.

• pytest VS vedro - a user suggested alternative 2 projects | 16 Jul 2023

Next, you need to install a testing framework that will be used for performing unit testing in your project. Several testing frameworks are available depending on the programming language used to create an application. For example, JUnit is commonly used for Java apps, pytest for Python apps, NUnit for .NET apps, Jest for JavaScript apps, and so on. We’ll use the Jest framework for this tutorial since we are using JavaScript.

Yea, read through the pytest docs.

• Testing an automation framework 2 projects | /r/softwaretesting | 16 May 2023

I absolutely agree about fixtures-as-arguments thing. Ward does this a lot better, using default values for the fixture factory. There's a long issue on ideas to implement something like that as a pytest plugin (https://github.com/pytest-dev/pytest/issues/3834), but it seems the resulting plugin relies on something of a hack.

• 2023 Development Tool Map 20 projects | dev.to | 19 Feb 2023

I recommend writing a few tests. py.test makes that quite simple:

Let look at an example of property-based testing using the Hypothesis library in Python:

Anecdotally, I had a fantastic experience with `clojure.spec.alpha` (with or without `test.check`), and when I went to use python's `hypothesis` it was just... abysmal. It seems like Hypothesis is unable to handle simple but "large" data sets >>by design<<, where "large" is really not so large. [0] It was such a pain that we ripped out Hypothesis (and generative testing altogether, sadly) completely from our python test suite at work. [0] https://github.com/HypothesisWorks/hypothesis/issues/3493

• Hypothesis 1 project | news.ycombinator.com | 1 Feb 2024

Hypothesis for Property-Based Testing: Hypothesis is a Python library facilitating property-based testing. It offers a distinct advantage by generating a wide array of input data based on specified properties or invariants within the code. The perks of Hypothesis include:

But then add tests! Tests for LaTeX equations that had never been executable as code. https://github.com/HypothesisWorks/hypothesis : > Hypothesis is a family of testing libraries which let you write tests parametrized by a source of examples. A Hypothesis implementation then generates simple and comprehensible examples that make your tests fail. This simplifies writing your tests and makes them more powerful at the same time, by letting software automate the boring bits and do them to a higher standard than a human would, freeing you to focus on the higher level test logic. > This sort of testing is often called "property-based testing", and the most widely known implementation of the concept is the Haskell library QuickCheck, but Hypothesis differs significantly from QuickCheck and is designed to fit idiomatically and easily into existing styles of testing that you are used to, with absolutely no familiarity with Haskell or functional programming needed.

pgregory.net/rapid is a modern Go property-based testing library initially inspired by the power and convenience of Python's Hypothesis.

• Was muss man als nicht-technischer Quereinsteiger in Data Science *wirklich* können? 1 project | /r/de_EDV | 13 Sep 2022

Hypothesis to generate dummy data for test.

To create your own test cases, I recommend you use hypothesis-graphql in combination with hypothesis. hypothesis is a property-based testing library. Property-based testing is an approach to testing in which you make assertions about the result of a test given certain conditions and parameters. For example, if you have a mutation that requires a boolean parameter, you can assert that the client will receive an error if it sends a different type. hypothesis-graphql is a GraphQL testing library that knows how to use hypothesis strategies to generate query documents.

The Hypothesis stateful testing code is somewhat self-contained, since it mostly builds on top of internal APIs that already existed.

What are some alternatives?

nose2 - The successor to nose, based on unittest2

Robot Framework - Generic automation framework for acceptance testing and RPA

Behave - BDD, Python style.

Slash - The Slash testing infrastructure

nose - nose is nicer testing for python

Schemathesis - Supercharge your API testing, catch bugs, and ensure compliance

mamba - The definitive testing tool for Python. Born under the banner of Behavior Driven Development (BDD).

The Python IDE for data science and web development

• Twitter Twitter
• Youtube Youtube

Pytest vs. Unittest: Which Is Better?

Python, being a versatile and widely used programming language, offers several testing frameworks to facilitate the testing process. Two prominent choices are pytest and unittest, both of which come with their own sets of features and advantages.

In this article, we’ll be covering the following sections:

• Introduction to each Python testing framework.
• Syntax and coding examples to give you a starting point.
• Advantages and disadvantages.
• A thorough comparison between the two testing frameworks.
• And finally, all the features that PyCharm provides for pytest and unittest.

This will help you determine which testing framework is the best for your needs.

Let’s get started.

Python Testing Frameworks

Python offers a variety of testing frameworks that cater to different needs and preferences. These frameworks facilitate the process of writing, organizing, and executing tests, ensuring the reliability and correctness of your code.

Two of the most popular Python testing frameworks are pytest and unittest. 

What is pytest?

Pytest is a popular and powerful testing framework for Python that simplifies the process of writing and executing tests. It is known for its simplicity, scalability, and ability to handle complex test scenarios while significantly reducing the boilerplate code required for testing.

In recent years, pytest has become one of the most popular choices for Python testing .

Features of pytest

• Fixture support: pytest provides a powerful fixture mechanism for setting up and tearing down resources needed for testing. This enhances test organization and readability.
• Parameterization: pytest allows easy parameterization of test functions, enabling the testing of multiple inputs without duplicating code. This enhances test coverage and maintains clean test code.
• Rich plugin architecture: pytest boasts a rich set of features and a vibrant plugin ecosystem. This extensibility allows customization and integration with other tools, making it adaptable to diverse testing needs.
• Concise syntax: The syntax for writing tests in pytest is concise and readable, making it easy for developers to adopt. This simplicity is conducive to rapid test development and maintenance.

How to write tests with pytest

Before writing tests with pytest, make sure you have it installed first. 

You can use Python Packages to install pytest in PyCharm:

Or you can also open your PyCharm terminal and write: 

Now, let’s take a simple example in which you’re creating an automated quiz.

In this program, we ask the user three questions and then return the number of answers they got correct. You can clone this exact project from this git repository.

We’ll be using this same example to create both pytest and unittest tests so you can see the difference in how they are structured.

Here’s how to write the test in pytest:

Overall, we have four test functions that test different cases. We are using mocking here to simulate different values a user can enter.

One important thing to note is that all test functions need to start with ‘test’ in pytest. If that’s not the case, then the pytest framework will fail to recognize it as a test function and won’t run it.

You can run each test separately or all of them together. 

Here’s the result:

Advantages and disadvantages of pytest

In this section, we will delve into the advantages of pytest, highlighting its strengths in areas apart from concise syntax, powerful fixtures, parameterization, and rich plugin support. Additionally, we’ll explore the potential drawbacks, acknowledging factors like a moderate learning curve for beginners, integration considerations, and nuances related to test discovery.

Advantages of pytest

1. Test discovery: pytest features automatic test discovery, eliminating the need for explicit configuration. This simplifies the testing process by identifying and executing test cases without manual intervention.

2. Skip/xfail tests: pytest provides the ability to skip or mark tests as expected failures using markers (@pytest.mark.skip and @pytest.mark.xfail). This flexibility is useful for excluding certain tests under specific conditions or marking tests that are known to fail.

3. Powerful assertion introspection: pytest provides detailed information on test failures, making it easier to diagnose and fix issues. The output includes clear and informative messages that help developers identify the exact cause of a failure.

4. Parallel test execution: pytest supports parallel test execution, allowing multiple tests to run concurrently. This feature can significantly reduce the overall test execution time, making it suitable for projects with large test suites.

5. Mocking and patching: pytest simplifies mocking and patching with built-in fixtures and libraries. This makes it easy to isolate components during testing and replace dependencies with controlled mocks.

6. Community support: pytest has a large and active community, leading to continuous development, regular updates, and a wealth of community-contributed plugins and resources. This vibrant community ensures that developers have access to extensive documentation and support.

7. Test isolation: pytest lets you run tests independently of each other, ensuring that the outcome of one test does not affect the outcome of another. This promotes cleaner, more robust, and maintainable test suites by ensuring that tests are reliable and easy to understand and debug.

These advantages collectively contribute to pytest’s popularity and effectiveness in various testing scenarios. Its flexibility, readability, and extensive feature set make it a valuable choice for both beginners and experienced developers alike.

While pytest is a powerful and widely adopted Python testing framework, like any tool, it has some potential disadvantages that users may encounter. It’s essential to consider these drawbacks when deciding whether pytest is the right choice for a particular project. 

Here are some potential disadvantages of pytest.

Disadvantages of pytest

1. Not in the Python Standard Library: Unlike unittest, which is part of the Python Standard Library, pytest is a third-party library. This means that for projects relying heavily on the Python Standard Library, there might be an additional step in installing and managing pytest.

2. Learning curve for beginners: For new developers, there might be a learning curve, especially if they are accustomed to other testing frameworks or if they are new to testing in general. Understanding and leveraging advanced features may take some time.

3. Integration with certain IDEs: While pytest integrates well with many IDEs, there can be occasional challenges in setting up the integration with some. Although this is not a widespread issue, it may require additional configuration in some cases.

4. Parallel test execution configuration: While pytest supports parallel test execution, configuring it for optimal performance might require additional setup and consideration. Developers need to understand and configure parallel execution settings correctly to avoid unexpected behaviors.

5. Slower test execution in some cases: In certain scenarios, pytest may exhibit slightly slower test execution compared to some other testing frameworks. This can be attributed to the additional features and flexibility provided by pytest, which may introduce some overhead.

6. Limited built-in fixtures for unittest-style setup/teardown: For developers accustomed to unittest-style setup and teardown using the setUp and tearDown methods, pytest might seem different. pytest encourages the use of fixtures, and while powerful, it might feel less intuitive for those transitioning from unittest.

7. Strictness in test discovery: pytest’s test discovery can be less strict compared to unittest in some cases. While this flexibility is an advantage for many, it might lead to unintended consequences if not carefully managed, especially in large and complex projects.

It’s important to note that many of these disadvantages are context-dependent, and what might be a drawback in one scenario could be an advantage in another.

Understanding the specific needs of a project and the preferences of the development team is crucial when evaluating whether pytest is the right testing framework for a given situation.

What Is unittest?

Unittest is a built-in testing framework in Python that follows the xUnit style and is inspired by Java’s JUnit. Since it’s part of the Python Standard Library, unittest provides a solid foundation for writing and organizing tests.

Features of unittest

• Test discovery: unittest automatically discovers and runs test cases without the need for explicit configuration. This simplifies test management and ensures all relevant tests are executed.
• Fixture support: unittest supports setup and teardown methods for test fixtures. While not as flexible as pytest’s fixtures, unittest’s fixtures provide the necessary functionality for resource management.
• Assertions: unittest comes with a variety of assertion methods for verifying expected outcomes. This includes methods like ‘assertEqual’, ‘assertTrue’, and more, ensuring comprehensive testing.
• Test suites: unittest allows grouping tests into test suites for better organization. This is beneficial for managing large codebases with numerous test cases.

How to write tests with unittest

We’ll be using the same quiz program example as before. This time, let’s test it with unittest.

Notice that in unittest, we don’t directly define our test functions. You need to define a test class that will contain all your test functions. In this case, it’s  class Test (unittest.TestCase). All unittest test classes need to be derived from unittest.TestCase, as this is the only way that the test functions will be invoked by the unittest framework.

Just like pytest, test functions in unittest also needs to start with ‘test’ in order for the framework to recognize and run it.

Here’s how to run the unittests individually or altogether in PyCharm:

Notice the difference between how unittest displays the test result vs how pytest does it. As you can see, unittest’s output is more summarized, whereas pytest displays test results for every individual test. 

Advantages and disadvantages of unittest

Unittest, the built-in testing framework in Python, offers a range of advantages that make it a solid choice for many developers and projects.

Here are some key advantages of unittest:

Advantages of unittest

• Community and industry standard: As a part of the xUnit family, unittest adheres to industry standards for unit testing frameworks. Developers familiar with xUnit-style frameworks in other languages will find unittest conventions familiar and consistent.
• Part of the Python Standard Library: unittest comes bundled with the Python Standard Library, ensuring its availability without the need for additional installations. This standardization promotes consistency across Python projects.
• Widely adopted and well-documented: As part of the Python Standard Library, unittest is widely adopted and well-documented. This makes it a familiar choice for developers transitioning from other languages with xUnit-style testing frameworks.
• Integration with IDEs: unittest integrates seamlessly with many integrated development environments (IDEs), providing features like test discovery, execution, and result visualization. This integration streamlines the testing workflow for developers using these tools.
• Standardized test output: unittest produces standardized test output, making it easy to integrate with other tools or continuous integration systems. The consistent output format enhances interoperability and ease of use.
• Consistent test isolation: unittest ensures consistent test isolation by creating a new instance of the test class for each test method. This prevents unintended side effects between different test cases.

While unittest may not have some of the advanced features found in third-party testing frameworks like pytest, its simplicity, standardization, and widespread adoption make it a reliable choice for many Python projects, particularly those where compatibility with the Python Standard Library is a priority.

However, there are some potential disadvantages to it, as well. It’s important to consider these limitations when deciding whether unittest is the most suitable choice for a particular project.

Disadvantages of unittest

1. Verbose syntax: One of the commonly mentioned criticisms of unittest is its verbose syntax. Compared to some other testing frameworks like pytest, writing test cases with unittest can require more lines of code, leading to a potential increase in boilerplate.

2. Limited parameterization support: unittest does not have native support for parameterized tests. 

3. Discovery requires rest prefix: By default, unittest requires test methods to be named with the prefix “test” for automatic discovery. While this convention is widely adopted, it can be restrictive if you want more flexibility in naming your test methods.

4. Learning curve for advanced features : While the basics of unittest are relatively easy to grasp, some of the more advanced features, such as creating custom test loaders or test runners, may have a more challenging learning curve.

5. Limited support for parallel test execution: unittest has limited built-in support for parallel test execution. Achieving parallelization may require additional effort and the use of external tools or libraries.

Despite these disadvantages, unittest remains a robust and reliable testing framework, particularly for projects that prioritize compatibility with the Python Standard Library and maintainability over advanced features provided by third-party frameworks.

Differences between pytest and unittest

The following table provides an overview of some key features and characteristics of both pytest and unittest.

Keep in mind that the suitability of a testing framework often depends on specific project requirements and team preferences.

Each framework has its strengths, and the choice between pytest and unittest should align with the testing needs and development philosophy of your project.

That said, for new projects with modern Python, pytest has become the default choice.

Pytest and unittest with PyCharm

Pytest in pycharm.

1. Intelligent code completion

PyCharm enhances the pytest experience with its intelligent code completion feature. As you write your test cases, PyCharm provides context-aware suggestions, making it easier to explore available fixtures, functions, and other elements related to pytest. This not only improves development speed but also helps you avoid typos and potential errors.

2. Test discovery and navigation

One of the key features of PyCharm is its robust support for test discovery. PyCharm automatically identifies and lists all your pytest tests, making it effortless to navigate through your test suite. You can quickly jump to the test definition, explore test fixtures, and view test outcomes directly from the editor. This streamlined workflow is particularly beneficial in large projects with extensive test suites. PyCharm also makes it very easy to run just one test, one file, or one directory, without knowing command line flags.

3. Integrated debugging

PyCharm provides seamless integration with pytest’s debugging capabilities. You can set breakpoints within your test code, run tests in debug mode, and step through the code to identify issues. The debugger interface in PyCharm offers a visual representation of the call stack, variable values, and breakpoints, facilitating efficient debugging and issue resolution.

4. Test result visualization

PyCharm presents test results in a clear and visually appealing manner. After running your pytest suite, you can easily navigate through the results, identify passed and failed tests, and view detailed information about each test case. This visual feedback helps developers quickly assess the health of their codebase and address any failures promptly.

5. Integration with version control systems

PyCharm seamlessly integrates with version control systems, ensuring that changes to both code and pytest tests are tracked consistently. This integration facilitates collaboration within development teams and helps maintain a unified versioning approach.

Unittest in PyCharm

1. Test discovery and execution

PyCharm provides robust support for discovering and executing unittest test cases. The IDE automatically identifies and lists all unittest tests in your project, simplifying test management. With just a few clicks, you can run individual test cases, entire test modules, or the entire test suite, ensuring a smooth and efficient testing process.

2. Code navigation and refactoring

PyCharm enhances the navigation and refactoring capabilities for unittest. Developers can easily navigate between test cases and corresponding code, making it straightforward to understand the relationship between tests and the tested code. Additionally, PyCharm supports various refactoring operations, allowing developers to modify their codebase confidently while maintaining the integrity of their test suites.

3. Code coverage analysis

PyCharm features a code coverage analysis tool that helps developers assess the extent to which their code is covered by unittests. This feature is valuable for identifying areas that may require additional testing, ensuring comprehensive test coverage.

4. Test configuration

PyCharm simplifies the configuration of unittest by providing an intuitive interface for adjusting testing parameters. Developers can customize test run configurations, specify test environments, and control other testing-related settings. This flexibility ensures that developers can adapt unittest to meet the specific requirements of their projects.

By using PyCharm, developers can seamlessly integrate both pytest and unittest into their workflow. PyCharm testing provides features like test discovery, execution, and result visualization for both frameworks.

How to choose between pytest vs. unittest?

Choosing between pytest and unittest depends on various factors, including project requirements, team preferences, and the complexity of testing scenarios. pytest is favored for its concise syntax, rich feature set, and flexibility, making it suitable for projects with diverse testing needs. 

On the other hand, unittest, being part of the standard library, provides simplicity and a consistent testing structure, making it a solid choice for projects that prioritize standardization and simplicity.

Ultimately, both pytest and unittest are capable testing frameworks, and the choice between them should align with your specific needs and development philosophy. Remember that the right testing framework can significantly contribute to the success of your project by increasing code quality and reliability.

Subscribe to PyCharm Blog updates

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy . I understand that I can revoke this consent at any time in my profile . In addition, an unsubscribe link is included in each email.

Thanks, we've got you!

Discover more

Learning Resources for pytest

Discover a trove of resources to help you master pytest, including guides from JetBrains and the community. Learn how PyCharm helps you when you’re working with pytest.

Three pytest Features You Will Love

Exploring better ways to test in Python? Find out how pytest can simplify your testing strategy in our comprehensive blog post.

Pytest with Eric

Pytest vs Unittest (Honest Review Of The 2 Most Popular Python Testing Frameworks)

In this article, we delve into the comparison between Unittest vs Pytest, shedding light on their strengths, weaknesses, and distinctive features.

Project Set Up

The project has the following structure

Getting Started

Prerequisites, test code example, testing — unittest.

Test miles to km conversion

Test km to miles conversion

Test unsupported input datatype- dict

Test unsupported input datatype- list

The following will be the output.

OK : When all the tests pass.

FAILED : When an AssertionError exception is raised (i.e. the assertion is unsuccessful), the output shows FAILED.

ERROR : When ALL the test cases did not pass and it raises an exception other than AssertionError .

As in the below example, it produces a Failed result when one test fails.

or an Error if there’s an error in the test

Testing — Pytest

Just like unittest , pytest give you a Fail message if your assertion fails .

However, it also gives you a Fail message, if there’s an error in the test , which in my opinion is not as clear as unittest .

Comparison Table — Unittest vs Pytest

About unittest, about pytest, recommendation - unittest vs pytest.

Unittest is a good choice for simple testing scenarios, PyTest is more powerful, flexible and extensible, making it a better choice for larger, more complex testing projects.

Additional Reading