The Old Bailey Voices data is the result of work I’ve done for the Voices of Authority research theme for the Digital Panopticon project. This will be the first of a few blog posts in which I start to dig deeper into the data. First I’ll review the general trends in trials, verdicts and speech, and then I’ll look a bit more closely at defendants’ gender. …
Posted at In Her Mind’s Eye
I’ve recently been working on the Digital Panopticon, a digital history project that has brought together (and created) massive amounts of data about British prisoners and convicts in the long 19th century, including several datasets which include heights for women. Adult height is strongly influenced by environmental factors in childhood, one of the most important being nutrition. So,
The height of past populations can thus tell historians much about the conditions that individuals encountered in their formative years. Given sufficient data it is possible to glimpse inside households in order to piece together a history of the impact that declining wages, rising prices, improvements in sanitation and diminishing family size had on mean adult stature.
However, many studies of height and nutrition in 18th- and 19th-century Britain focused on military records and therefore had little to say about women. The turn to using the rich records of heights for men and women (and children) in 19th-century penal records has been more recent.
Today’s post is going to look at height patterns in four Digital Panopticon datasets, mainly using a kind of visualisation that many historians aren’t familiar with: box plots. If you’ve seen them and not really understood them, it’s OK – I didn’t have a clue until quite recently either! And so, I’ll start by attempting to explain what I learned before I move on to the actual data.
A box plot, or box and whisker plot, is a really concentrated way of visualising what statisticians call the “five figure summary” of a dataset: 1. the median average; 2. upper quartile (halfway between the median and the maximum value); 3. lower quartile (halfway between the median and minimum value); 4. minimum value; and 5. maximum value.
Here’s a diagram:
The thick green middle bar marks the median value. The two blue lines parallel to that (aka “hinges”) show the upper and lower quartiles. The pink horizontal lines extending from the box are the whiskers. In this version of a box plot, the whiskers don’t necessarily extend right to the minimum and maximum values. Instead, they’re calculated to exclude outliers which are then plotted as individual dots beyond the end of the whiskers.
So what’s the point of all this? Imagine two datasets: one contains the values 4,4,4,4,4,4,4,4 and the other 1,3,3,4,4,4,6,7. The two datasets have the same averages, but the distribution of the values is very different. A boxplot is useful for looking more closely at such variations within a dataset, or for comparing different datasets, which might look pretty much the same if you only considered averages.
These are the four datasets:
For each dataset, I only included women who had a year of birth, or whose year of birth could be calculated using an age and date, as well as a height. (I say “heights” above because I can’t guarantee that they are all unique individuals; but nearly all of them should be.) In all the following charts I’m including only adult women aged over 19.
Here’s what happens when you plot the heights for each birth decade in RHC.
(This is generated using the R package ggplot2 , and it looks a little bit different from many examples you’ll see online because ggplot has a nice feature to vary the width of the boxes according to the size of the data group.)
The first thing I look for is incongruities that might suggest problems with the data, and on the whole it looks good – the boxes are mostly quite symmetrical and none of the outliers is outside the realms of possibility (the tallest woman is 74.5 inches, or 6 foot 2 1/2, and the shortest is 48 inches), though I’m slightly doubtful that there were women born in the 1800s in this dataset, which gets going in the 1880s; still, they’re a very small number so unlikely to skew things much overall. Since the data seems to be OK on first sight, the interesting thing to note here is that from the 1850s onwards, the women are getting taller, and those born in the 1890s are quite a lot taller than the 1880s cohort. This is fairly consistent with Deb Oxley’s (more fine-grained) observations of the same data.
Here’s CIN:
Again, we have a reasonable spread of heights and fortunately very small number of slightly questionable early births. (It happens to be the case that this data was manually transcribed, whereas RHC was created using Optical Character Recognition – but on the other hand, the source for RHC was printed and much more legible than the handwritten indents.) Ignoring for now the very small groups before the 1770s, the tallest decade cohort of women in this data is those born in the 1790s and thereafter they get consistently shorter.
Let’s put all four datasets together! (click on the image for a larger version)
I’ve filtered out women born before 1750 and after 1899, because the numbers were very small, and some extreme outliers (more about those later…). Then I added a guideline at the median for the 1820s (the mid-point), as I think it helps in seeing the trends.
It might seem surprising at first that the late 18th-century women of HCR are taller than any subsequent cohorts until the 1890s. Yet the trends here are broadly consistent with the pioneering research by Roderick Floud et al on British men and boys between 1740 and 1914. They argued “that the average heights of successive birth cohorts of British males increased between 1740 and 1840, fell back between 1840 and 1850, and increased once again from the 1850s onwards” (Harris, ‘Health, Height and History’). The British population was less well-fed for much of the 19th century (as food resources struggled to keep up with rapid population growth), and it got smaller as a result. Our women’s growth after 1850 may be slower than for the men (until the 1890s) though; perhaps it took longer for women than men to start growing again.
Finally, though, I have to put in a big caveat about the HCR data. I mentioned that I excluded some extreme outliers from the chart above. HCR was by far the worst offender, and if you look closely at the 18th-century cohorts covered by HCR, the boxes aren’t quite as symmetrical as the 19th-century ones. If we visualise it using a histogram (another handy one for examining the distribution of values in a dataset), we can see more clearly that there’s something up. A ‘normal’ height distribution in a population should look like a “bell curve” – quite tightly and symmetrically clustered around the average. CIN and RHC are close:
But this is what HCR looks like. This is not good.
If we’re lucky, much of the problem could turn out to be errors in the data which can be fixed. After all, it’s at least roughly the right kind of shape! The big spike at 60 inches (5 feet) rings plenty of alarm bells though. It looks reminiscent of a problem we have with much of the age data in the Digital Panopticon, known as “heaping“, a tendency to round ages to the nearest 0 or 5 (people often didn’t know their exact dates of birth). The age heaping is very mild in comparison to this spike, so I think it could well be another issue with either the transcription or the method used to extract heights. But if it turns out that’s not the case, this could be pretty problematic. We’re assuming the prisoners were properly measured, but we don’t know anything about the equipment used. For all we know, it might often have been largely guess work. In the end, we might find that HCR simply isn’t reliable enough to use for demographic analysis. There’s very little height data for women born in the 18th century, so this is a potentially really important source. But what if it’s not up to the job?
Further reading
John Canning, Statistics for the Humanities (2014), especially chapter 3.
Introduction to Statistics: Box plots
H Maxwell-Stewart, K Inwood and M Cracknell, ‘Height, Crime and Colonial History’, Law, Crime and History (2015).
Deborah Oxley, David Meredith, and Sara Horrell, ‘Anthropometric measures of living standards and gender inequality in nineteenth-century Britain’, Local Population Studies, 2007.
Deborah Oxley, Biometrics, http://www.digitalpanopticon.org (2017).
Bernard Harris, ‘Health, Height, and History: An Overview of Recent Developments in Anthropometric History’, Social History of Medicine (1994).
Jessica M. Perkins et al, ‘Adult height, nutrition, and population health’, Nutrition Reviews (2016).
This is a version of the paper I gave at the Digital Panopticon launch conference at Liverpool in September 2017.
In the interests of fostering reproducible research in the humanities, I’ve put all the data and R code underlying this paper online on Github – details of where to find them are at the end.
Defendants’ voices are at the heart of the Digital Panopticon Voices of Authority research theme I’ve been working on with Tim Hitchcock. We know that defendants were speaking less in Old Bailey Online trials as the 19th century went on; we’ve tended to put this in the context of growing bureaucratisation and the rise of plea bargaining.
I want to think about it slightly differently in this paper though. The graph above compares conviction/acquittal for defendants who spoke and those who remained silent, in trials containing direct speech between 1781 and 1880. It suggests that for defendants themselves, their voices were a liability. This won’t surprise those who’ve read historians’ depiction of the plight that defendants found themselves in 18th-century courtrooms without defence lawyers, in the “Accused Speaks” model of the criminal trial (eg Langbein, Beattie).
But this isn’t a story of bureaucrats silencing defendants (or lawyers riding in to the rescue). I want to suggest that, once defendants had alternatives to speaking for themselves (ie, representation by lawyers and/or plea bargaining), they made the choice to fall silent because it was often in their best interests.
The Old Bailey Corpus, created by Magnus Huber, enhanced a large sample of the OBP 1720-1913 for linguistic analysis, including tagging of direct speech and tagging about speakers. [In total: 407 Proceedings, ca. 14 million spoken words, ca. 750,000 spoken words/decade.]
Trials with multiple defendants have been excluded from the dataset because of the added complexity of matching the right speaker to utterances (and they aren’t always individually named in any case). [But of course this begs the question of whether the dynamics and outcomes of multi-defendant trials might be different…]
Trial outcomes have also been simplified; if there are multiple verdicts or sentences only the most “serious” is retained. Also, for this paper I include only trials ending in guilty/not guilty verdicts, omitting a handful of ‘special verdicts’ etc.
Caveat!
Working assumption is that nearly all silent defendants do have a lawyer and the majority of defendants who speak, don’t.
Sometimes, especially in early decades, defendants had a lawyer and also spoke. Unfortunately, the OBC tagging doesn’t distinguish between prosecution and defence lawyers, and not all lawyer speech was actually reported.
But, more seriously, is it safe to assume that ‘silent’ defendants were really silent? Occasionally defendant speech was actually censored in the Proceedings (in trials where other speech was reported), eg a man on trial for seditious libel in 1822 whose defence “was of such a nature as to shock the ears of every person present, and is of course unfit for publication”. But that was a very unusual, political, case. (See t18220522-82 and Google Books, Trial of Humphrey Boyle)
[However, it was suggested in questions after the presentation that maybe the issue isn’t so much total censorship as in the case above, but that the words of convicted defendants might be more likely to be partially censored, which would problematise analyses that centre on extent and content of their words. This could be a particular problem in 1780s and 1790s; maybe less so later on.]
So work to be done here – eg, look at trials with alternative reports specifically to consider defendants’ words.
Start with some broad context.
The number of cases peaked during the 1840s and dramatically fell in the 1850s. (Following the Criminal Justice Act 1855, many simple larceny cases were transferred to magistrates’ courts.)
Percentage climbs from 1780s (in 1778 Proceedings became near-official record of court), peaks early 19th c and then after major criminal justice reforms of late 1820s swept away most of the Bloody Code, shown by red line, substantial fall in proportion of trials containing speech.
This was primarily due to increase in guilty pleas, which were previously rare. After the reforms, 2/3 of trials without speech are guilty pleas.
(Ignore the spike around 1792, due to censorship of acquittals.) Gradual increase in conviction rates which declines again after mid 19th c.
But if we exclude guilty pleas and look only at jury trials, the pattern is rather different.
Conviction rates in jury trials after the 1820s rapidly decrease – not much over 60% by end of 1870s. That’s much closer to 18th-century conviction rates (when nearly all defendants pleaded not guilty), in spite of all the transformations inside and outside the courtroom in between.
Here the green line is the Prisoners’ Counsel Act of 1836, which afforded all prisoners the right to full legal representation. But the smoothed trend line indicates that it had no significant impact on defendant speech. Defendants had, at the judge’s discretion, been permitted defence counsel to examine and cross-examine witnesses since the 1730s.Legal historians emphasise the transformative effect of the Act; but from defendants’ point of view it seems less important; for them it was already a done deal and the Bloody Code reforms were much more significant.
This breaks down the first graph by decade – shows that the general pattern is consistent throughout period, though exact % and proportions do vary.
Moreover, harsher outcomes for defendants who speak continues into sentencing. Pleading guilty (though bear in mind this only really applies to c.1830-1880, whereas silent/speaks bars are for whole period) most likely to result in imprisonment, much less likely to receive transportation (and hardly ever death) sentence. Defendants who speak are the most likely to face tougher sentences – death or transportation, more so than the silent.
(Don’t yet have actual punishments – the next big job is getting the linked Digital Panopticon life archives…)
How much did defendants say? Not a lot. The largest single group of defendants is the silent (ie, 0 words). But even those who spoke usually didn’t say very much. [average overall was 55 words] Eloquent, articulate defendants few and far between!
So if you did speak, it was better to say plenty!? Or in other words, more articulate defendants had a better chance of acquittal (though they were still slightly worse off than the silent).
Finish with focus on defendants’ defence statements – made by nearly all defendants who spoke and for the majority the only thing they did say (a minority questioned witnesses or made statements at other points in the trial).
overall word counts of defence statements * guilty (n=7696) average wc 44.97 * notguilty (n=1414) average wc 65.15
On average, defence statements by the acquitted were longer. Again highlights that more articulate defendants do better.
Also, there is more variety (less repetition) in the statements of acquitted defendants. 98% (1374) of their 1414 defence statements are unique (crudely measured, as text strings). Whereas 93.17% (7170) of statements by convicted defendants are unique.
Start to look more closely at what they say? Not possible yet to investigate in depth, but use some simple linguistic measures.
| mercy |
| picked |
| man |
| i |
| distress |
| carry |
| along |
| them |
| beg |
| stop |
| up |
| young |
In linguistics, keywords are “items of unusual frequency in comparison with a reference corpus”. Compared the larger set of defence statements by defendants who were convicted with defence statements by defendants who were acquitted
Table above is the words least likely to be associated with acquittal – ie, the least successful defence statements…
I want to highlight:
Remember that many defence statements were not really ‘defences’; they were more of an appeal to the judges’ clemency after sentencing (‘I beg for mercy’) or claiming extenuating circumstances (‘I was in distress’) in particular. Also Playing down offence – ‘I picked up the things’.
And in general many short bare statements beginning with “I” rather than more complex narratives.
So I picked four of the most frequent short (non-)defences that are heavily associated with convictions, to explore a bit further. (excludes use of any of these within longer defences)
| defence | frequency | % convicted |
|---|---|---|
| nothing to say | 109 | 98.17 |
| mercy | 125 | 98.40 |
| picked up/found | 223 | 93.72 |
| distress | 82 | 97.56 |
Main variants:
The next four graphs show the percentage of defendant speakers who use each phrase in short defence statements in each decade.
This was very popular before 1810s – peaks at use by 4% of defendants who speak in decade 1801-10 and then rapidly disappears.
Slightly later popularity – slower decline after 1810s
Less dramatic decline after 1820s.
Curious that this doesn’t appear at all in 1780s; peaks 1810s.
So there are variations in timing/speed of decline, but broadly, these hopeless ’non-’defence statements, which are almost certain to be followed by conviction, are all declining in use and rarely heard in the courtroom after the 1820s. That fits, it seems to me, with both the gradual decline in defendant speech and the more rapid rise from the late 1820s of plea bargaining.
First, the defence lawyer option meant that defendants were better off finding the money for a lawyer who could try to undermine the prosecution case through aggressively examining witnesses. This was happening from the 1780s onwards.
And second, the plea bargaining option from the late 1820s meant that if defendants really had no viable defence, had been caught red-handed, they were better off pleading guilty in return for a less harsh punishment.
And so: for defendants who wanted to walk free or at least lessen their punishment, if not for later historians trying to hear their voices and understand what made them tick, silence was golden.
We’re holding an afternoon workshop on record/data linkage in Sheffield on 4 November. The aim is to explore the challenges and rewards of applying automated nominal record linkage to large-scale historical datasets, with all their variability, fuzziness and uncertainties, but we’d also very much welcome participants and insights from all fields concerned with data linkage including social sciences, health sciences and computer science. In addition to presentations about our work in progress on 90,000 19th-century prisoners and convicts, we have guest speakers who will bring extensive experience of historical record linkage projects to the discussion. It’s free to attend and anyone with an interest, at any stage of their academic career, is welcome (I’d particularly love to see plenty of PhD students!). More info can be found on our website here (and there’s also a programme to download).
Record linkage is really at the heart of the Digital Panopticon project’s goals to explore the impact of the different types of punishments on Old Bailey Online defendants between about 1780 and 1875 (along with working on data visualisations for exploring, presenting and communicating the data and research findings). Our research questions include: How can we improve current record-linkage processes to maximise both the number of individuals linked across different datasets and the amount of information obtained about each individual? What is the minimum amount of contextual information needed in order to conduct successful large-scale record linkage of data pertaining to specific individuals?
I’ve blogged in the past about problems associated with historical record linkage where you don’t have handy unique IDs (like, say, National Insurance numbers): names are often crucial but are highly problematic, and problems with a source like Old Bailey Online that tells us about sentences but not actual punishments. Those are among our biggest headaches with Digital Panopticon.
There are a lot of missing people when we link OBO to transportation records, and a lot of possible reasons for linking to fail. There might be errors in the data created at almost any point between the making of the original source and our production of a specific dataset to feed to the computer: eg, if you’re extracting a London-only subset from a national dataset and you’re not careful, you might also end up with records from Londonderry. Oops. (“You” there is an euphemism for “I”. )
Then there are problems caused by spelling variations in names, or the use of aliases and different names. And the problem of common names. As I blogged before: “How do you decide whether one Robert Scott is the same person as another Robert Scott, or someone else altogether?” But that gets much worse when the name in question is “Mary Smith”.
And the fails that are due to the gaps in our data: Were they pardoned? Did they die in prison or on the hulks before they could be transported? And so we are on a quest to track down sources that can tell us these things and fill the gaps (not all of which have been digitised; some of which have probably not even survived, especially from the 18th century).
Irreconcilable conflicts can emerge between different sources (eg, different trial dates and places). At this point we have to turn to the specialist knowledge of the project team on how, when and where particular sources were created so we can attempt to rate the relative reliability of two conflicting sources. But how are we going to handle those weightings when we’re dealing with thousands of people and the links are all probables anyway? (Just because source A is generally more reliable for a certain piece of information than source B doesn’t mean A is always right and B is always wrong if they’re in conflict.)
So there will be plenty to discuss at the workshop and for the next three years!
For tasters of what we’ve been getting up to so far:
The view from my seat at the DP data visualisation workshop
Yesterday, I went to All Souls College, Oxford, for a data visualisation workshop organised by the Digital Panopticon project.
The project – a collaboration between the Universities of Liverpool, Sheffield, Oxford, Sussex and Tasmania – is studying the lives of over 60,000 people sentenced at the Old Bailey between 1780 and 1875, to look at the impact of different penal punishments on their lives.
It aims to draw together genealogical, biometric and criminal justice datasets held by a variety of different organisations in Britain and Australia to create a searchable website that is aimed at anyone interested in criminal history – from genealogists to students and teachers, to academics.
This is a huge undertaking, and it is no wonder that the project aims to harness digital technologies in making the material accessible to a wide audience. But how could…
View original post 530 more words
Starting a new project is exciting and intensely busy (which is also my excuse for taking a month to blog about it). And the Digital Panopticon is the biggest one we’ve done yet.
‘The Digital Panopticon: The Global Impact of London Punishments, 1780-1925’ is a four-year international project that will use digital technologies to bring together existing and new genealogical, biometric and criminal justice datasets held by different organisations in the UK and Australia in order to explore the impact of the different types of penal punishments on the lives of 66,000 people sentenced at The Old Bailey between 1780 and 1925 and create a searchable website.
The Panopticon, for anyone who doesn’t know, was a model prison proposed by the philosopher Jeremy Bentham (1748-1832): “a round-the-clock surveillance machine” in which prisoners could never know when they were being watched. In Bentham’s own words: “a new mode of obtaining power of mind over mind, in a quantity hitherto without example”. Although Bentham’s plan was rejected by the British government at the time, there were later prisons built along those lines (Wikipedia), and the panopticon has become a modern symbol of oppressive state surveillance and social control.
Bentham criticised the penal policy of transportation and argued that confinement under surveillance would prove a more effective system of preventing future offending. One of DP’s basic themes is to test his argument empirically by comparing re-offending patterns of those transported and imprisoned at the Old Bailey. But it will go further, to compare the wider social, health, generational impacts of the two penal regimes into the 20th century.
Technically, DP brings together a number of different methods/techniques we’ve worked on in various projects over the years: digitisation, record linkage, data mining and visualisation, impact, connecting and enhancing resources, with the goal of developing “new and transferable methodologies for understanding and exploiting complex bodies of genealogical, biometric, and criminal justice data”.
However, it’s a much more research-intensive project than the ones we’ve done recently, and that’s reflected in the depth and breadth of the seven research themes. These are based on three central research questions/areas:
What’s also exciting (and new for us) is that we’ll have PhD students as well as postdoc researchers (adverts coming soon). Lots of PhD students! Two are part of the AHRC funding package – one at Liverpool and one at Sheffield – and the partner universities have put up funding for several more (two each at Liverpool and Sheffield and one at Tasmania, I think).
The first at Sheffield has just been advertised and the deadline is 2 December (to start work in February 2014):
The Social and Spatial Worlds of Old Bailey Convicts 1785-1875
The studentship will investigate the social and geographical origins and destinations of men and women convicted at the Old Bailey between 1785 and 1875, in order to shed light on patterns of mobility and understandings of identity in early industrial Britain. Using evidence of origins from convict registers and social/occupational and place labels in the Proceedings, the project will trace convicts from their places of origin through residence and work in London before their arrests, to places of imprisonment and subsequent life histories. Analysis of the language they used in trial testimonies will provide an indication of how identities were shaped by complex backgrounds.
Spread the word – and watch this space (and the project website) for more announcements soon!
PS: the project is on Twitter: follow at @digipanoptic
