stesiam

Statistics of Statistics’ Graduates

stesiam — Sun, 23 Jul 2023 00:00:00 GMT

Introduction

Note

Frequent updates should be expected until this note be erased.

The recent study guide includes data about the graduates and their average grade of graduation over the years. First and foremost the data we are interested in are included in table form (which is good 😀) but it is part of a pdf file (which is not good 😢). Thankfully, tabulizer is the solution to this kind of problems, as it one of its various features is to extract tabular data from pdf files.

Load Packages

As I will extract tables from a pdf file, I will definitely need tabulizer package. Unfortunately, I was not able to install the specific package, as I was getting an error similar to this one. This issue seems to be related to rJava package and this comment solved the issue. After installing rJava, I was able to install successfully tabulizer as below:

#remotes::install_github(c("ropensci/tabulizerjars", "ropensci/tabulizer"))

library(dplyr)
library(tidyr)
library(highcharter)
library(reactable)
library(rJava)
library(tabulapdf)
library(pdftools)

Extract Data

The study guide gives a general description of the university, as well as the prerequisites for a degree and a detailed description of each course. In total, the guide is a little bit less than 200 pages! Of course we don’t need My main source of data is the Department’s study guide. The most recent one (2022) has data on admissions, graduations etc., since 2004. The study guide gives a general description of the university, as well as the prerequisites for a degree and a detailed description of each course. In total, the guide is a little bit less than 200 pages! Of course we don’t need everything in there. I am just interested on the tables of the last pages, so I will extract those pages first from the original pdf.

# url = "https://www.unipi.gr/faculty/mbouts/anak/OS_22_23.pdf"
# 
# download.file(url, 
#               destfile = "sg22.pdf",
#               method = "wget",
#               extra = "--no-check-certificate")

pdf_subset('sg22.pdf',
  pages = 186:190,  output = "subset.pdf")

So, we extracted the pages which we are interested in. Let’s take a look at them:

Thankfully, tabulizer comes with a very handy function to extract all the tables from a PDF file. Yeap, I know the original study guide is written in greek but don’t worry that’s just to take a basic understanding of how the pdf and the tables look. I will translate the column names when I will work with the data. But look at the bright side. At least we will get a notebook with ggplot2 visualizations as an alternative of those Excel graphs :)

statistics_tables <- extract_tables(
    file   = "subset.pdf", 
    method = "decide", 
    output = "tibble")

New names:
New names:
New names:
New names:
New names:
• `` -> `...1`

Extracting the tables from splitted PDF I get a list of 5 tables with my data. It’s amazing that in a matter of seconds I get all the information in a format ready for analysis. If I were to write them the traditional way (copy-paste) it would definitely take me an hour.

Admitted students

So, in Greece there is 1 standard way to be admitted to a university. Although there are 3 more ways which require some certain conditions. I will try to explain them as simple as possible.

Main Exams

Once a year, third-year high school students from all over Greece take exams on the same subjects at the same time. The exams are known as Panhellenic Exams. Until today, it’s one of the few things in Greece that as its integrity is not disputed, as the papers of students are getting graded by teachers from other areas. However, it has also faced considerable criticism for the pressure it places on students. In my opinion a fair one as everything in your life is depending on these exams… If you fail you should wait to retake them next year.

Typically the exams are being held between the second half of May and the first week of June. The students’ grades are getting published approximately either the end of June or the first days of July. Then you are completing a list on which you are declare which departments you are interested to. On the end of July the minimum grades to be admitted for each department are announced. Those can fluctuate significantly every year as they are depending on both students’ performance on the Exams and the difficulty of the exams.

In a nutchell, the departments from big cities like Attica (Athens, Piraeus) and Thessaloniki have the biggest demand and so the minimum grades for those are higher from the rest ones. For example, the Statistics Department in Piraeus had a minimum grade of 11700 in 2019 (for simplicity consider it like 11.7/20). The corresponding department of Statistics in University of Aegean the same year had a minimum grade of 5100 (5.1/20) (yes, that’s not a typo). Well there are many reasons behind that, as the continuance of austerity in Greece, but in general that’s the pattern.

Not so fun fact but when I made my list of preference for studies, Statistics in Piraeus was something like 15th place, so I guess my fate was that. Okay and a little bit of anxiety. :)

Transfer

As I wrote earlier there are some exceptions. First of all the Admission by Transfer is referring to transfer your place in one department with one similar-study in other city. There are many criteria mainly based on your income. For example, a student admitted on Statistics on University of Aegean could be admitted on Statistics Department on Piraeus (i.e. in case his/her family hasn’t enough income).

These seats are limited.

Entry Exams

In case you have already graduated a Bachelor programme then you are able to give Entry Exams on your department of choice, instead of the nightmare of Panhellenic Exams.

admitted_students = statistics_tables[[1]] %>%
  .[-1,] %>%
  setNames(c("Year", "Main_exams", "Transfer", "Entry_exams", "Other", "Total")) %>%
  tidyr::drop_na() %>%
  dplyr::filter(Transfer >= 0) %>%
  select(-Total)

I would like to examine the percentage of students who have been admitted by the other 3 ways over the years.

Student Population

f = statistics_tables[[2]] %>%
  setNames(c("Year", "BSc", "MSc_A","MSc_B", "PhD")) %>%
  drop_na() %>%
  mutate(MSc_B = stringr::str_replace(MSc_B, "-", "0")) %>%
  mutate(across(-Year, ~ as.integer(.))) %>%
  mutate(MSc = MSc_A + MSc_B)

highchart() %>%
  hc_chart(type = "area") %>%
  hc_title(text = "Stacked Area Chart Example") %>%
  hc_xAxis(categories = f$Year) %>%
  hc_yAxis(title = list(text = "Population (millions)")) %>%
  hc_plotOptions(area = list(stacking = "normal", marker = list(enabled = FALSE))) %>%
  hc_add_series(name = "Bachelor", data = f$BSc) %>%
  hc_add_series(name = "Master", data = f$MSc) %>%
  hc_add_series(name = "PhD", data = f$PhD) %>%
  hc_legend(enabled = TRUE)

latest_data <- f %>%
  filter(Year == "2022-2023") %>%
  select(-Year) %>%
  tidyr::pivot_longer(cols = everything(), names_to = "continent", values_to = "population") %>%
  dplyr::filter(continent %in% c("BSc", "MSc", "PhD"))

highchart() %>%
  hc_chart(type = "treemap") %>%
  hc_title(text = "Interactive Population by Continent in 2017") %>%
  hc_add_series(
    data = list_parse(
      latest_data %>%
        mutate(
          name = continent,
          value = population
        )
    ),
    layoutAlgorithm = "squarified"
  ) %>%
  hc_tooltip(pointFormat = "{point.name}: {point.value}")

Master Students

highchart() %>%
  hc_chart(type = "area") %>%
  hc_title(text = "Stude") %>%
  hc_xAxis(categories = f$Year) %>%
  hc_yAxis(title = list(text = "# of students")) %>%
  hc_plotOptions(area = list(stacking = "normal", marker = list(enabled = FALSE))) %>%
  hc_add_series(name = "Applied Statistics", data = f$MSc_A) %>%
  hc_add_series(name = "Actuarial Science", data = f$MSc_B) %>%
  hc_legend(enabled = FALSE) %>%
  hc_title(text = "Total population of students") %>%
  hc_subtitle(text = "The total number of postgraduate students enrolled on the department's two programmes.")

PhD Students Ratio

# statistics_tables[[3]] %>%
#   setNames(c("Year", "MSc_AppliedStats", "MSc_Actuar")) %>%
#   slice(-c(1:2))

ms_phd_students = statistics_tables[[2]] %>%
  setNames(c("AcademicYear", "BSc", "MSc (AppliedStat)", "MSc (Actuar)", "PhD")) %>%
  .[-1,]

rownames(ms_phd_students) = 1:nrow(ms_phd_students)

Warning: Setting row names on a tibble is deprecated.

ms_phd_students = ms_phd_students %>%
  mutate_at(c("BSc", "MSc (AppliedStat)", "MSc (Actuar)", "PhD"), as.numeric) %>%
  mutate(pct = round((PhD/BSc)*100, 2)) %>%
  mutate(AcademicYear = stringr::str_remove(AcademicYear, "-.*"))

Warning: There was 1 warning in `mutate()`.
ℹ In argument: `MSc (Actuar) = .Primitive("as.double")(`MSc (Actuar)`)`.
Caused by warning:
! NAs introduced by coercion

Structure of Students

B = statistics_tables[[2]] %>%
  setNames(c("Year", "BSc_students", "MScStudentsA", "MScStudentsB", "PhD")) %>%
  .[-1,] %>%
 mutate_at(., vars(-Year), as.integer) %>%
 mutate(
   MScStudentsB = tidyr::replace_na(MScStudentsB, 0),
   MSc_Students = MScStudentsA + MScStudentsB
 ) %>%
  dplyr::select(-c("MScStudentsA", "MScStudentsB")) %>%
  relocate(., MSc_Students, .after = "BSc_students") %>%
  mutate(
    total = BSc_students + MSc_Students + PhD,
    BSc_students = BSc_students/total,
    MSc_Students = MSc_Students/total,
    PhD = PhD/total
  ) %>%
  mutate_at(vars(!c(Year, total)), ~round(.*100, 2)) %>%
  dplyr::select(c(-total)) %>%
  tidyr::pivot_longer(., cols = !c(Year), values_to = "Obs")

Warning: There was 1 warning in `mutate()`.
ℹ In argument: `MScStudentsB = .Primitive("as.integer")(MScStudentsB)`.
Caused by warning:
! NAs introduced by coercion

Graduation Rate

Graduates & Graduation Grade

I hope you are still here because for the end I hold the best part. Finally, how did we perform? The study guide gives a distribution

extractGraduatesAndGrades = statistics_tables[[4]] %>%
  setNames(c("Year", "[5.0 - 6)", "[6, 7)", "[7-8.5)", "[8.5-10]", "AVG_Grade")) %>%
  slice(-c(1:4))  %>%
  mutate(across("AVG_Grade"), separate(.,AVG_Grade, into = c("AVG_Grade", "Graduates"),sep = "\\(")) %>%
  select(c("Year", "AVG_Grade", "Graduates")) %>%
  mutate(
    "Graduates" =  stringr::str_remove_all(Graduates, "\\)")
  ) %>%
  mutate(
    AVG_Grade = stringr::str_replace_all(AVG_Grade, ",", ".")
  ) %>%
  mutate_at(vars("AVG_Grade", "Graduates"), as.numeric)

Online vs On-campus Exams

Due to pandemic COVID-19, Greek government suspended the operation of schools (elementary, high school, universities) on Tuesday 10 March 2020. Some days later imposed lockdown measures. That period our department organized online lectures and organized online exams for the majority of the courses on Spring Semester and on the September (retake exams). Long story short that lasted for at least 2 years.

Students Graduated in 6 Years

Last but not least, maybe the most nervewrecking statistic is that one. Just for clarification, our studies have a duaration of four years. Although the majority will not successfully finish their degree even in 6 Years. This is important as currently has been voted the time-limit on BSc studies. We have 6 years to complete our studies otherwise we will not take the degree. In the next years, the best case scenario is that 60% of my collegeaus who were about to graduate will not get their degree.

Acknowledgements

Dataset based on recent study guide recent study guide of Department of Statistics and Insurance Science of the University of Piraeus.

Image by Victoria Loveland from Pixabay

Citation

BibTeX citation:

@online{2023,
  author = {, stesiam},
  title = {Statistics of {Statistics’} {Graduates}},
  date = {2023-07-23},
  url = {https://stesiam.com/posts/eda-graduates-of-statistics/},
  langid = {en}
}

For attribution, please cite this work as:

stesiam. (2023, July 23). Statistics of Statistics’ Graduates. Retrieved from https://stesiam.com/posts/eda-graduates-of-statistics/

Accessing API of data.gov.gr

stesiam — Tue, 04 Jul 2023 00:00:00 GMT

Introduction

In traditional data analysis, the analyst is usually asked to “cleanse” the data, to give it a suitable format in order to be analysed. This process implies that the data exists in concentrated form in a file. However, this is not always the case. The problem with analysing a dataset is that it does not take into account any new values. In this case I have to re-download the new data and rerun the analysis through R or whatever tool I’m using. This workflow is not productive. Where it becomes apparent the inefficiency of the above process is when there is a constant flow of data. In these cases the usefulness of the API is shown since it gives us the new values with relatively little effort. At the same time the use of the API favours the integration of new data, so if we build a predictive model its accuracy will remain at a satisfactory level. Finally, another positive contribution is in Shiny Apps in order to automate data analysis and the information is reliable and up-to-date for the visitor.

In general there are many APIs from which we can get important information. For more details about the availability of free APIs you can see a relevant repository with a list of them. In this article we will deal with Greek Government’sAPI available through data.gov.gr.

Requesting an API Key

However, like most APIs, this one requires us to register with the platform. We can do this on the relevant page by filling in our details. All the fields of the form must be completed, as shown below:

Preview request page for access to the API

Next, you should check your e-mail because you will get a message with a Token through which the API will be used. Be sure to also check the spam folder. By the way, in case you lose the token and have deleted the mail and reapply (with the same mail) it will be resent to your email.

Using API

Once we receive the Token, we need to somehow get the data. On the specific platform there are two ways.

Using API from the website:
Using API with R

One way (and probably the least efficient) is to request the data directly from the data.gov.gr website. This is on the one hand extremely simple, but on the other hand we are downloading a fixed version of the data and therefore if I want to update the data I have to download the data again.

Just a use-case

Before completing this article, I decided it would be helpful to provide an example. At the time of writing, there are 49 databases to choose from. To demonstrate the usefulness of the API, I will select a database that is updated fairly frequently. One such dataset is that of passengers travelling on ships.

library(httr)
library(jsonlite)

I set the base url of my data. Given the fact that I am interested on sailing traffic I will use the respective endpoint:

base = "https://data.gov.gr/api/v1/query/sailing_traffic"

As the documentation points out, we need to set the date range we are interested in. It is worth noting that you cannot retrieve a large range with a single API call. Returning to the previous example, ship passenger traffic data starts in 2017 and goes up to the present day (2023). Let’s assume that we don’t mind that, and we want the data from the first four days of July in 2023.

date_from = "2023-07-01"
date_to = "2023-07-04"

API_URL = paste0(base, "?date_from=", date_from, "&", "date_to=", date_to)

call = httr::GET(url = API_URL,
    add_headers(`Authorization` = paste0('Token token_id')
    )
)

Where it refers to token_id, you should insert the token that was sent to you by data.gov.gr via email. So, after requesting the data through a GET request and waiting a bit, we receive a list with various pieces of information. What we are interested in is the data, so we look into the content category of the list. However, we notice that the information is in an unreadable format since it is in hexadecimal form.

data = base::rawToChar(call$content)

Using rawToChar converts the response into readable characters but I still need to convert my data to tabular format in order to begin with my analysis.

#data = jsonlite::fromJSON(data, flatten = T)

Finally, using the jsonlite package, I receive a data frame named as data which includes every destination, passenger/car count for the requested date.

Acknowledgements

Image by Christopher Kuszajewski from Pixabay

Statistics Studies in Greece

stesiam — Tue, 20 Jun 2023 00:00:00 GMT

Introduction

Σε αυτό το άρθρο θα ήθελα να συγκεντρώσω όλες αυτές τις διάσπαρτες πληροφορίες που υπάρχουν για τα τμήματα Στατιστικής στη χώρα μας. Θα αναλύσω ως επί το πλείστον πληροφορίες που μπορούν να βρεθούν στο διαδίκτυο σαν να είχα να συμπληρώσω μηχανογραφικό. Ωστόσο, σε κάθε περίπτωση σας αποτρέπω από το να επιλέξετε σχολή βασίζοντας την επιλογή σας σε μόνο αυτό το άρθρο. Σε αυτή τη κατηγορία εντάσσω απλά όσες σχολές σχετίζονται με τη Στατιστική. Τα τμήματα που θα συγκρίνω είναι τα εξής:

Department of Statistics of Athens University of Economics and Business
Department of Statistics and Insurance Science of University of Piraeus
Τμήμα Στατιστικής και Αναλογιστικών - Χρηματοοικονομικών Μαθηματικών του Πανεπιστημίου Αιγαίου
Department of Statistics of University of Western Macedonia

library(readxl)
library(stringr)
library(dplyr)
library(sysfonts)
library(reactable)
library(showtext)
library(reactablefmtr)
library(highcharter)
library(here)

[1] FALSE

Minimum admission scores

Ένα πρώτο κριτήριο για αρκετούς είναι η βάση εισαγωγής. Αν και κατά τη γνώμη μου αυτός ο δείκτης δείχνει κατά πόσο είναι επιθυμητή αυτή η σχολή και όχι το κατά πόσο καλή ή κακή είναι αυτή. Παρακάτω, συγκέντρωσα ιστορικά στοιχεία βάσεων (από το 2013 μέχρι το 2022) από τις τέσσερις σχολές Στατιστικής και έκανα ένα απλό διάγραμμα. Σε αυτό φαίνεται ότι ιστορικά το τμήμα με τη μεγαλύτερη βάση είναι του ΟΠΑ (με μόνη εξαίρεση το 2015). Δεύτερο σε προτίμηση έρχεται το τμήμα Στατιστικής του Πειραιά. Επιπλέον, τα υπόλοιπα δύο τμήματα Στατιστικής (Αιγαίου και Δυτικής Μακεδονίας) ταυτίζονται αρκετά ως προς τη βάση εισαγωγής τους.

title_text = "Βάσεις εισαγωγής σε τμήματα Στατιστικής"
subtitle_text = "Η βάση εισαγωγής είναι ο μικρότερος αριθμός μορίων - βαθμολογίας που πρέπει να συγκεντρώσει κάποιος προκειμένου να εισαχθεί στη συγκεκριμένη σχολή. Διαχρονικά, οι σχολές με τη μεγαλύτερη βάση είναι του Οικονομικού Πανεπιστημίου Αθηνών και του Πανεπιστημίου Πειραιά. Τέλος, παρατηρούμε ότι όλες 
 οι  σχολές Στατιστικής έχουν σχετικά χαμηλή βάση εισαγωγής."
caption_text = "Πηγή:Βάσεις Πανελληνίων
 stesiam, 2024"

hchart(tmimata_statistikis, "line", hcaes(x = Year, y = Vasi, group = University)) |>
  hc_title(
    text = title_text,
    align = "left") |>
  hc_xAxis(title = "Έτος") |>
  hc_yAxis(title = "Βάση εισαγωγής") |>
  hc_subtitle(text = subtitle_text, align = "left")

Figure 1: Βάσεις εισαγωγής σχολών Στατιστικής (2013 - 2022)

Θέσεις

Κάθε χρόνο το Υπουργείο Παιδείας ανακοινώνει τις διαθέσιμες θέσεις για κάθε σχολή και για κάθε κατηγορία εξεταζόμενων. Σε αυτή την ανάλυση ασχολούμαι με τους υποψήφιους των Γενικών Λυκείων που αποτελούν και τη συντριπτική πλειοψηφία των συμμετεχόντων. Με μία πρώτη ματιά παρατηρώ μία αντιστροφή των αποτελεσμάτων σε σύγκριση με τη βάση εισαγωγής. Δηλαδή, στις σχολές με την σχετικά υψηλότερη βάση παρέχονται λιγότερες θέσεις, ενώ οι περισσότερες θέσεις διατίθενται στις σχολές χαμηλότερης βάσης.

title_text = "Θέσεις εισαγωγής σε τμήματα Στατιστικής (ΓΛ90)"
subtitle_text = "Oι θέσεις για όσους θέλουν να φοιτήσουν στο τμήμα Στατιστικής του ΟΠΑ φαίνεται να είναι αρκετά περιορισμένες για τους υποψήφιους των Γενικών Λυκείων προσφέροντας λίγες παραπάνω από 100. Τα τμήματα Αιγαίου και Πειραιά τουλάχιστον τα τελευταία χρόνια ταυτίζονται ως προς τους πόσους θα δεχτούν (από τη κατηγορία ΓΛ90) με το τμήμα του Πειραιά να μειώνει σημαντικά τους εισακτέους του, σε σύγκριση με το 2013."

caption_text = " Πηγή: Βάσεις Πανελληνίων
 stesiam, 2024"


hchart(tmimata_statistikis, "line", hcaes(x = Year, y = Theseis, group = University)) |>
  hc_title(
    text = title_text,
    align = "left") |>
  hc_xAxis(title = "Έτος") |>
  hc_yAxis(title = "Θέσεις εισαγωγής") |>
  hc_subtitle(text = subtitle_text, align = "left")

Figure 2: Availability of admission seats on Statistics Departments (2013 - 2022)

Βέβαια, η παροχή των θέσεων δεν συνεπάγεται πάντοτε και πλήρωση αυτών. Πολλές από αυτές μπορεί να μείνουν αδιάθετες λόγω χαμηλής ζήτησης. Αυτό το φαινόμενο παρατηρείται σε σχολές εκτός των αστικών κέντρων. Τα τελευταία χρόνια υπάρχει και ένας επιπρόσθετος λόγος μιας και έχει θεσμοθετηθεί η ελάχιστη βάση εισαγωγής (ΕΒΕ), η οποία αποτρέπει την βάση μιας σχολής να πέσει κάτω από έναν συγκεκριμένο βαθμό (π.χ. κάτω από 7 ή 13) και για αυτό παρατηρείται στο πρώτο διάγραμμα των βάσεων μία απότομη αύξηση στις σχολές του Αιγαίου και της Δυτικής Μακεδονίας. από την εφαρμογή της να δούμε και άλλες αδιάθετες θέσεις. Έτσι, ο αριθμός των εισακτέων μπορεί πλέον να διαφέρει σημαντικά όχι απλώς μεταξύ τμημάτων εντός και μεγάλων αστικών κέντρων (π.χ. Πανεπιστήμιο Αιγαίου, Πανεπιστήμιο Δυτικής Μακεδονίας - Γρεβενά), αλλά και ως προς τον διαθέσιμο αριθμό εισακτέων και των εγγεγραμμένων φοιτητών.

years <- unique(tmimata_statistikis$Year)

# Step 3: Prepare series data
series_data <- tmimata_statistikis %>%
  group_by(University) %>%
  summarise(data = list(Theseis)) %>%
  mutate(name = University) %>%
  select(name, data) %>%
  list_parse()


highchart() %>%
    hc_chart(type = "area") %>%
    hc_title(text = "Distribution of available places by Department") %>%
    hc_subtitle(text = "Το τμήμα Στατιστικής του Πειραιά είναι αυτό που προσέφερε περισσότερες θέσεις αναλογικά με τις συνολικά προσφερόμενες σε πανελλήνιο επίπεδο. Πλέον το νεοσύστατο τμήμα των Γρεβενών έχει τη μεγαλύτερη προσφορά θέσεων.") %>%
    hc_xAxis(categories = years, tickmarkPlacement = "on", title = list(enabled = FALSE)) %>%
    hc_yAxis(title = list(text = "Percent (%)"), labels = list(format = "{value}%")) %>%
    hc_tooltip(shared = TRUE, pointFormat = "{series.name}: {point.percentage:.1f}% ({point.y:,.0f})
") %>%
    hc_plotOptions(area = list(
        stacking = "percent",
        lineColor = "#ffffff",
        lineWidth = 1,
        marker = list(lineWidth = 1, lineColor = "#ffffff")
    )) %>%
    hc_add_series_list(series_data)

Figure 3: Proportion of seats offered in Statistics departments(2013 - 2022)

Requirements to Graduate

Και τα τέσσερα τμήματα προσφέρουν 4ετή φοίτηση και κάποιος ορίζεται ως πτυχιούχος με την απόκτηση 240 ECTS. Οι διαφορές τους ανάγονται κυρίως στο πλήθος των μαθημάτων ή στο αν υπάρχει προϋπόθεση για πτυχιακή. Σημειώνεται ότι οι παραπάνω πληροφορίες βασίζονται στους πρόσφατους οδηγούς σπουδών των παραπάνω τμημάτων (2022-2023).

Το UniPi2011 αναφέρεται στο πρόγραμμα σπουδών του τμήματος Στατιστικής του Πειραιά που ίσχυε για τους φοιτητές που μπήκαν στο τμήμα από το 2011 μέχρι το 2016. Οι εισακτέοι του τμήματος από το 2017 ακολουθούν το νέο πρόγραμμα σπουδών του 2017 (UniPI2017)

highchart() %>%
  hc_chart(type = "bar") %>%
  hc_title(text = "Course structure of Statistics departments") %>%
  hc_subtitle(text = "The number of courses required to obtain an undergraduate degree from public statistics departments in Greece. The composition of the programs is also analyzed based on the flexibility of the curriculum (number of electives as a percentage of required courses).")  %>%
  hc_caption(text = "stesiam, 2025") %>%
  hc_xAxis(
       categories = degree_requirements$Images,
         labels = list(useHTML = TRUE),
         title = list(text = "")
   ) %>%
  hc_yAxis(title = list(text = "Number of courses to graduate")) %>%
  hc_plotOptions(series = list(stacking = "normal")) %>%
  hc_add_series(name = "Electives", data = degree_requirements$Elective, stack = "courses") %>%
  hc_add_series(name = "Obligatory", data = degree_requirements$Comp, stack = "courses")

Figure 4: Required courses for obtaining a degree per department

Επιλεγόμενα μαθήματα

Παραπάνω είδαμε σε πόσα μαθήματα θα πρέπει να εξεταστεί ένας σπουδαστής του αντίστοιχου τμήματος για να πάρει πτυχίο. Πόση ελευθερία υπάρχει σε κάθε τμήμα προκειμένου ο ίδιος ο φοιτητής να προσαρμόσει τις σπουδές του στις προτιμήσεις του και τα προσωπικά του ερευνητικά ενδιαφέροντα; Αυτό προσπάθησα να μελετήσω με την προσθήκη της μεταβλητής των ελεύθερων μαθημάτων και του ποσοστού που αποτελούν αυτά ως προς τα συνολικά μαθήματα.

degree_requirements = degree_requirements |>
  arrange(-Electivepct)

highchart() %>%
  hc_chart(type = "bar") %>%
  hc_title(text = "Flexibility of Study Guides") %>%
  hc_subtitle(text = "Ο αριθμός των επιλεγόμενων μαθημάτων ως ποσοστό των συνολικών μαθημάτων που απαιτούνται για το πτυχίο.")  %>%
  hc_caption(text = "stesiam, 2025") %>%
 hc_xAxis(
       categories = degree_requirements$Images,
         labels = list(useHTML = TRUE),
         title = list(text = "")
   ) %>%
  hc_yAxis(title = list(text = "Percentage of electives/total courses")) %>%
  hc_plotOptions(series = list(stacking = "normal")) %>%
  hc_add_series(name = "Electives", data = degree_requirements$Electivepct, stack = "courses")

Figure 5: The percentage of electives out of the total number of courses that are required to obtain a degree.

Late graduation

undergrads_by_year <- read_excel(here::here(work_dir, "undergrads_by_year.xls")) %>%
  dplyr::select(c(3,4,25)) %>%
  setNames(c("Dept", "StudentsUnEq6", "studentsOv6")) %>%
  tidyr::drop_na(Dept) %>%
  .[-c(1),] %>%
  mutate(across(c(StudentsUnEq6, studentsOv6), ~ as.integer(.))) %>%
  mutate(totalStudents = StudentsUnEq6 + studentsOv6,
         pctOv6 = studentsOv6 / totalStudents,
         nonPctOv6 = 1 - pctOv6) %>%
  arrange(-pctOv6) %>%
  dplyr::filter(str_detect(Dept, "ΣΤΑΤΙΣΤΙΚΗ"))

highchart() %>%
  hc_chart(type = "bar") %>%
  hc_title(text = "Percentage of students that have exceeded n+2 years") %>%
  hc_subtitle(text = "Το τμήμα του Πειραιά έχει το μεγαλύτερο ποσοστό καθυστερημένης αποφοίτησης με 7 στους 10 φοιτητές του τμήματος να αποτελούν φοιτητές 7ου ή μεγαλύτερου έτους. Με μεγάλη ποσοστιαία διαφορά ακολουθούν τα τμήματα της Αθήνας και της Σάμου. Το τμήμα των Γρεβενών είναι νεοσύστατο και δεν έχει φοιτητές ν+2.") %>%
  hc_xAxis(categories = undergrads_by_year$Dept) %>%
  hc_yAxis(title = list(text = "Value")) %>%
  hc_plotOptions(series = list(stacking = "normal")) %>%
  hc_add_series(name = "Until 6 Years", data = undergrads_by_year$nonPctOv6) %>%
  hc_add_series(name = "Over 6 Years", data = undergrads_by_year$pctOv6) %>%
  hc_yAxis(
    title = list(text = "Value"),
    plotLines = list(
      list(
        color = "orange",
        width = 2,
        value = 0.5,
        dashStyle = "ShortDash",
        label = list(
          y = 200,
          text = paste0("Average: ", round(50, 1)),
          align = "left",
          style = list(color = "orange", fontWeight = "bold", fontSize = 10)
        )
      )
    )
  ) %>%
  hc_tooltip(
  valueDecimals = 2
)

Figure 6: Barplot of late graduation rates by department

Other Information

reactable::reactable(others,
                     columns = list(

    Uni = colDef(name = "Πανεπιστήμιο"),
    Diloseis = colDef(name = "Όριο δηλώσεων", 
                      style = function(value) {
      if (value == "Όχι") {
        color <- "#008000"
      }  else {
        color <- "#e00000"
      }
      list(color = color, fontWeight = "bold")
    }
),
    Alysides = colDef(name = "Αλυσίδες μαθημάτων",
                      style = function(value) {
      if (value == "Όχι") {
        color <- "#008000"
      }  else {
        color <- "#e00000"
      }
      list(color = color, fontWeight = "bold")
    }
  ),
    Ptyxiaki = colDef(name = "Πτυχιακή",
                      style = function(value) {
      if (value == "Όχι") {
        color <- "#008000"
      }  else {
        color <- "#e8c010"
      }
      list(color = color, fontWeight = "bold")
    }),
    Vathmos =  colDef(name = "Υπολογισμός βαθμού")
  ))

Table 1: Characteristics of study guides

Mapping Departments

Finally, it would be interesting to practice my skills in creating maps with R. I usually use Shapefiles to have a base map (in this case, the boundaries of Greece) or a package to define them, such as the rnaturalearth accompanied by the rnaturalearthdata και rnaturalhires packages. This was my approach in the first version of my article. Fortunately, in the past period, I have made a great effort to transform my static plots into interactive ones using highcharter. That’s what I am also trying to do with maps, as the aforemntioned library offers relevant mapping tools. On our specific example, we need Greece’s boundaries which are available from the library (as are the boundaries of other countries).

students_by_dept = undergrads_by_year |>
  dplyr::filter(stringr::str_detect(Dept, "ΣΤΑΤΙΣΤΙΚΗ")) %>%
  mutate(Short = c("UniPi", "AUEB", "Aegean", "Grevena"))

uni_coord = data.frame(
  uni = c("UniPi", "Aegean", "AUEB", "UoWM"),
  unilong = c("University of Piraeus", "University of Aegean",
              "Athens University of Economics and Business",
              "University of Western Macedonia"),
  dept = c("Statistics and Insurance Science", "Statistics and Actuarial Science",
           "Statistics", "Statistics"),
  long = c(23.6529793, 26.5664138,23.7300928, 21.4565181),
  lat = c(37.77, 39.0851185, 38.1940201, 40.1197471),
  img = c("https://upload.wikimedia.org/wikipedia/en/thumb/a/a4/UNIPI_Emblem.png/250px-UNIPI_Emblem.png", "https://upload.wikimedia.org/wikipedia/en/d/df/Aegean_University_logo.gif", "https://upload.wikimedia.org/wikipedia/en/thumb/8/89/AUEB_EMBLEM.png/250px-AUEB_EMBLEM.png","https://upload.wikimedia.org/wikipedia/commons/thumb/4/42/UoWM.png/120px-UoWM.png"),
  established = c(1938, 1984, 1920, 2003),
  dpt_est = c(1977,2000, 1989, 2019),
  town = c("Piraeus", "Samos", "Athens", "Grevena")
)

uni_coord$symbol <- c(
  "url(https://upload.wikimedia.org/wikipedia/commons/e/e4/Map_marker_icon_%E2%80%93_Nicolas_Mollet_%E2%80%93_Pin.svg)",
  "url(https://upload.wikimedia.org/wikipedia/commons/e/e4/Map_marker_icon_%E2%80%93_Nicolas_Mollet_%E2%80%93_Pin.svg)",
  "url(https://upload.wikimedia.org/wikipedia/commons/e/e4/Map_marker_icon_%E2%80%93_Nicolas_Mollet_%E2%80%93_Pin.svg)",
  "url(https://upload.wikimedia.org/wikipedia/commons/e/e4/Map_marker_icon_%E2%80%93_Nicolas_Mollet_%E2%80%93_Pin.svg)"
)

# Load Greece Map
hcmap("countries/gr/gr-all", name = "SA") %>%
hc_add_series(
  data = uni_coord,
  type = "mappoint",
  hcaes(lon = long, lat = lat, name = uni),
  name = "Departments",
  color = "brown4",
  marker = list(
    symbol = "url(https://cdn-icons-png.flaticon.com/512/5081/5081368.png)",  # example map pin SVG
    width = 20,
    height = 20
  )
) %>%
  hc_title(text = "Statistics Departments of Greece") %>%
  hc_subtitle(text = "In Greece, there are a total of four public undergraduate statistics schools. The duration of study at these schools is four years. They are located in Athens, Piraeus, Samos, and Grevena.") %>%
  hc_caption(text = "stesiam, 2025") |>
  hc_tooltip(
  useHTML = TRUE,  # Enables HTML formatting
  formatter = JS("
    function() {
      return '' +
      '' + 
               '' + this.point.unilong + '
(' + this.point.uni + ')
' + 
               '
' +
               '' + this.point.dept + '
' +
                '
' +
               '
' +
               'Town:' + this.point.town + '
' +
               'Est. University: ' + this.point.established + '
' + 
               'Est. Dept.: ' + this.point.dpt_est + '
' + 
               'Undergrads: ' + this.point.studs +
               '
';
    }
  ")
)  %>%
  hc_legend(enabled = FALSE)

Επίλογος

Η προσωπική μου άποψη είναι ότι το τμήμα Στατιστικής του Αιγαίου είναι μία πολύ δελεαστική επιλογή. Παρουσιάζεται (το τονίζω αυτό, παρουσιάζεται) ως ένα σοβαρό τμήμα με έναν σύγχρονο οδηγό σπουδών (λίγα μαθήματα που αντισταθμίζεται με τον ανάλογο φόρτο εργασίας). Το συγκεκριμένο τμήμα κατά τη γνώμη μου αδικείται από τη βάση που έχει. Αν είχα την επιλογή να διαλέξω δίχως να λάβω υπόψιν το οικονομικό κριτήριο (κόστος ενοικίου κτλ.) θα ήταν η πρώτη μου επιλογή. Βέβαια, αυτή η επιλογή όπως έγραψα και στην αρχή βασίζεται αποκλειστικά σε πράγματα που θα έβρισκα στο διαδίκτυο και απλά τα μάζεψα σε ένα άρθρο. Υπάρχουν και άλλα πράγματα που έχουν σημαντικότατο ρόλο, όπως οι διδάσκοντες, το επίπεδο μαθήματος, τήρηση κανονισμών, εγκαταστάσεις, προσβασιμότητα και άλλα που δυστυχώς θα τα μάθετε, όταν θα είναι πολύ αργά.

Acknowledgements

Image by WOKANDAPIX from Pixabay

Citation

BibTeX citation:

@online{2023,
  author = {, stesiam},
  title = {Statistics {Studies} in {Greece}},
  date = {2023-06-20},
  url = {https://stesiam.com/posts/statistics-studies-in-greece/},
  langid = {en}
}

For attribution, please cite this work as:

stesiam. 2023. “Statistics Studies in Greece.” June 20, 2023. https://stesiam.com/posts/statistics-studies-in-greece/.

Kaggle’s Greek Community

stesiam — Sat, 06 May 2023 00:00:00 GMT

Introduction

Kaggle is one of the most well-known communities of data analysts/scientists with over 10 million active users (Heads or Tails, 2020). Besides that, Kaggle offers an abundance of functionalities (Notebooks), information (through Discussions between users) and Competitions. It is worth noting that there are other similar communities but they cannot compare to the full functionality of Kaggle. For example, DrivenData could be considered an alternative for participating in ML competitions, but it neither provides the possibility to create notebooks nor has a large number of users.

Kaggle’s logo

Kaggle Machine Learning & Data Science Survey is an annual survey conducted by Kaggle. The platform asks its users to analyze users’ data in the context of a competition. In this notebook, I conduct an analysis based on 2021’s survey in order to compare Greek data analysts with the rest of the world.

Prerequisites

Import libraries

This notebook will definitely make some charts, so the ggplot2 package is necessary. Also, having variables with too many values (e.g. country of each Kaggle user) is an indication of using tables, and for this the reactablefmtr package will help to get a nice result.

# General purpose R libraries
library(readr)
library(dplyr)
library(tidyr)
library(forcats)
library(gridExtra)
library(countrycode)

# Tables
library(kableExtra)
library(reactablefmtr)

# Graphs
library(ggplot2)
library(ggtext) # Add support for HTML/CSS on ggplot
library(showtext)
library(sysfonts) # System / Google fonts
library(glue)
library(ggflags)

# Other R packages
library(fontawesome)
#library(htmltools) # for building div/links

# Other settings
options(digits=4) # print only 4 decimals
options(warn = -1)

## Load fonts

# font_families_google() ## see list with available Google fonts

font_add_google(name = "Lilita One", family = "title", db_cache = F)
font_add_google(name = "Ysabeau Office", family = "subtitle", db_cache = F)
font_add_google(name = "Spline Sans", family = "text", db_cache = F)

showtext_auto()
showtext::showtext_opts(dpi = 300)

library(highcharter)

Import data

Using read.csv() from readr package, I import my dataset and I name it as kaggle_2021. The dataset includes in the first line the question which is not required for my data analysis, so I exclude it from my dataset.

kaggle_2021 = read_csv("data/kaggle_survey_2021.csv")

# Delete second line
kaggle_2021 = kaggle_2021[-c(1),]

Prepare Data

Since my analysis is based on Greek users, I split the dataset into two parts. One part includes exclusively Greek users and all the rest another. Thus, we can observe any differences or similarities with broader Kaggle’s userbase.

# Recoding Q2 variable

kaggle_2021$Q2 = kaggle_2021$Q2 %>%
  fct_recode(
    "Other" = "Nonbinary",
    "Other" = "Prefer not to say",
    "Other" = "Prefer to self-describe"
  )

## Recoding kaggle_2021$Q3

kaggle_2021$Q3 <- kaggle_2021$Q3 %>%
  fct_recode(
    "Hong Kong" = "Hong Kong (S.A.R.)",
    "Other" = "I do not wish to disclose my location",
    "Iran" = "Iran, Islamic Republic of...",
    "UAE" = "United Arab Emirates",
    "UK" = "United Kingdom of Great Britain and Northern Ireland",
    "USA" = "United States of America",
    "Vietnam" = "Viet Nam"
  )

## Recoding kaggle_2021$Q6
kaggle_2021$Q6 <- kaggle_2021$Q6 %>%
  fct_recode(
    "0 years" = "I have never written code"
  )

## Recoding kaggle_2021$Q4
kaggle_2021$Q4 <- kaggle_2021$Q4 %>%
  fct_recode(
    "Bachelor" = "Bachelor’s degree",
    "PhD" = "Doctoral degree",
    "Other" = "I prefer not to answer",
    "Master" = "Master’s degree",
    "No" = "No formal education past high school",
    "ProfDoc" = "Professional doctorate",
    "UniNoDegree" = "Some college/university study without earning a bachelor’s degree"
  )

kaggle_2021$Q4 <- kaggle_2021$Q4 %>%
  fct_relevel(
    "No", "UniNoDegree", "Bachelor", "Master", "PhD", "ProfDoc",
    "Other"
  )


kaggle_2021_compare = kaggle_2021 %>%
  mutate(Q3 = if_else(Q3 != "Greece", "Other", Q3))

Kaggle’s community

One of the first thing I observed when I signed up on Kaggle was the vast majority of nationalities and the multicultural origin of the platform. Many people from many countries all in one platform gathered sharing the same passion for Data Science and Data Analytics. Something like Facebook but for Statistics :)

Nevermind, I decided to make a reactable to see from which nationalities the platform is comprised from. One out of four users are from India which makes them the most populous nation in platform. By the way, Greek users are way less making a 0.39% of Kaggle’s userbase.

Assumptions Note

We should note that the results are from Kaggle’s Survey. That is to say from people that participated. An assumption has to be done that the distribution of the users that participated is the same with the ones who didn’t.

a = kaggle_2021 %>%
  group_by(Q3) %>%
  summarise(n = n()) %>%
  mutate(pct = round(n/nrow(kaggle_2021) * 100, digits = 2)) %>%
  arrange(desc(pct)) %>%
            reactable(.,
              defaultPageSize = 6,
              theme = espn(),
              columns = list(
                Q3 = colDef(name = "Country"),
                n = colDef(name = "Population", 
                           defaultSortOrder = "desc"),
                pct = colDef(name = "Percentage (%)")),
              defaultColDef = colDef(
                cell = data_bars(data = ., 
                                 text_position = "outside-base",
                                 fill_opacity = 0.5, round_edges = T
                )
              )
            ) %>%
 add_title(
    title = reactablefmtr::html("Kaggle's Community by Country "),
    margin = reactablefmtr::margin(t=0,r=0,b=3,l=0)
    )  %>% 
            add_subtitle("India is the most represented country, followed by the United States.", font_size = 14, font_weight = "normal") %>% 
            add_source("stesiam | Data : Kaggle Survey 2021", font_size = 12, align = "right")

a

Kaggle's Community by Country

India is the most represented country, followed by the United States.

stesiam | Data : Kaggle Survey 2021

Women participation in DS

Generally, women are under-represented in labor market. According to the World Bank (2023), only one in two women participates in the labor market, in contrast to men whose corresponding participation is 7 in 10. Does DS community follow the same pattern? As it seems, it varies.

data = kaggle_2021 %>%
  group_by(Q3) %>%
  summarise(n = n(),
            Women = sum(factor(Q2) == "Woman"),
            pct_women = Women/n *100) %>%
  dplyr::filter(Q3 != "Other")

data$iso2c <- countrycode(data$Q3, "country.name", "iso2c")
data$iso2c = tolower(data$iso2c)

theme_set(theme_light())
data %>%
  ggplot() +
  geom_col(aes(x = pct_women, y = reorder(Q3, pct_women), fill = pct_women)) +
  scale_fill_gradient2(low="purple", high="purple4")+
  geom_label(data = subset(data, pct_women == max(pct_women) | Q3 == "Greece" | pct_women == min(pct_women)), aes(x = pct_women - 1.5, y = reorder(Q3, pct_women), label = paste0(round(pct_women, digits = 1), "%")), family = "text", size = 2.3) +
 ggflags::geom_flag(x = 0.3, aes(y = Q3,
                         country = iso2c), 
                         size = 4) + 
  geom_vline(xintercept = mean(data$pct_women), linetype = "dashed", color = "pink1") +
  geom_text(aes(x= mean(data$pct_women)+1, label=paste0("Average: ", round(mean(data$pct_women), digits = 2), " %"), y = "Colombia"), angle=270,
family = "text") +
  scale_x_continuous(limits = c(0,50)) +
  labs(
    title = "Women participation in DS community per country",
    subtitle = glue("Based on Kaggle's 2021 Survey women are underrepresented in DS. The country with the 
 highest women participation is Tunisia and Peru is one with the lowest. Last but not least,
 **Greece** has a relatively disappointing rate of women participation holding 15th place with 
 15.7\\%, given the fact that the average is {round(mean(data$pct_women),digits = 2)} %"),
    caption = "**Data:** Kaggle Survey 2021
stesiam, 2023",
    x = "Percentage (%) of Women",
    y = "Countries"
  ) +
  coord_cartesian(expand = FALSE)

library(purrr)
data_decreasing = data %>% dplyr::arrange(-pct_women)

# Add emoji flags
data_decreasing <- data_decreasing %>%
    mutate(flag = map_chr(iso2c, ~ {
        chars <- utf8ToInt(toupper(.x))
        paste0(
            intToUtf8(127397 + chars[1]),
            intToUtf8(127397 + chars[2])
        )
    }))


highchart() %>%
    hc_chart(type = "bar", inverted = TRUE) %>%
    hc_title(text = "Women participation in DS community per country") %>%
    hc_subtitle(text = glue("Based on Kaggle's 2021 Survey women are underrepresented in DS. The country with the highest women participation is Tunisia and Peru is one with the lowest. Last but not least, Greece has a relatively disappointing rate of women participation holding 15th place with 15.7%, given the fact that the average is {round(mean(data$pct_women),digits = 2)} %")  ) %>%
    hc_caption(text="Data: Kaggle Survey 2021 | stesiam, 2023") %>%    hc_xAxis(categories = data_decreasing$Q3, title = list(text = NULL)) %>%
    hc_yAxis(title = list(text = "Percent (%)")) %>%
    hc_legend(enabled = FALSE) %>%
    hc_add_series(
        name = "Women %",
        data = map(seq_len(nrow(data_decreasing)), function(i) {
            list(
                y = data_decreasing$pct_women[i],
                Women = data_decreasing$Women[i],
                n = data_decreasing$n[i],
                country = data_decreasing$Q3[i],
                flag = data_decreasing$flag[i]
            )
        })
    ) %>%
    hc_tooltip(
        useHTML = TRUE,
        formatter = JS("
      function() {
        var point = this.point;
        return '' + point.flag + ' ' +
               '' + point.country + '
' +
               'Percent: ' + Highcharts.numberFormat(this.y, 2) + '%
' +
               'Women: ' + point.Women + ' / ' + point.n;
      }
    ")
    )

Age Distribution

Greece’s Kaggle Community it is comprised from more elerly people comapred to the rest of Kaggle’s community. More specifically, Greece’s most prevalent age group is 25-29 and a sufficient proportion of users on their 40s. On the contrary, Kaggle’s community is quite youthy with most prevalent the three youngest age groups. Aggrevetating, those groups consist six out of ten userbase of Kaggle’s community.

theme_set(theme_light())

data1 = kaggle_2021_compare %>%
  select(Q3, Q1) %>%
  group_by(Q3,Q1) %>%
  summarise(n = n()) %>%
  group_by(Q3) %>%
  mutate(total = sum(n),
         pct = round(n/total *100, digits = 1)) %>%
  select(Q3, Q1, pct)

`summarise()` has grouped output by 'Q3'. You can override using the `.groups`
argument.

library(highcharter)

highchart() |>
  hc_chart(type = "areaspline") |>
  hc_title(text = "Age Distribution of Kaggle Community (🇬🇷 / 🌍)") |>
  hc_xAxis(categories = unique(data1$Q1)) |>
  hc_yAxis(title = list(text = "Population (%)")) |>
  hc_tooltip(shared = TRUE, valueSuffix = " units") |>
  hc_plotOptions(
    areaspline = list(
      fillOpacity = 0.5  # match the Highcharts demo's fill
    )
  ) |>
  hc_legend(
    layout = "vertical",
    align = "left",
    verticalAlign = "top",
    x = 450,
    y = 100,
    floating = TRUE,
    borderWidth = 1
    ) |>
  hc_add_series(
    name = "Greece",
    data = data1 %>% dplyr::filter(Q3 =="Greece") %>% pull(pct)
  ) |>
  hc_add_series(
    name = "Rest of the World",
    data = data1 %>% dplyr::filter(Q3 =="Other") %>% pull(pct)
  )

Educational Background

data3 = kaggle_2021_compare %>%
  select(Q3, Q4) %>%
  group_by(Q3, Q4) %>%
  summarise(n = n()) %>%
  group_by(Q3) %>%
  mutate(pct = round(n/sum(n)*100, digits = 1))  %>%
  ungroup()

`summarise()` has grouped output by 'Q3'. You can override using the `.groups`
argument.

d = data3 %>% dplyr::filter(Q4 %in% c("Bachelor", "Master", "PhD"))

highchart() %>%
  hc_chart(type = "bar") %>%
  hc_title(text = "Age Distribution of Kaggle Community (🇬🇷 / 🌍)") %>%
  hc_subtitle(text="Greek Kagglers have attained higher studies.") %>%
  hc_caption(text="stesiam, 2023") %>%
  hc_xAxis(categories = d$Q4,
           title = list(text = NULL)) %>%
  hc_yAxis(min = 0,
           title = list(text = "Percentage (%) of total respondents by Country / Region", align = "high"),
           labels = list(overflow = "justify")) %>%
  hc_tooltip(valueSuffix = " units") %>%
  hc_plotOptions(bar = list(dataLabels = list(enabled = TRUE))) %>%
  hc_series(
    list(name = "Greece", data = d %>% dplyr::filter(Q3 =="Greece") %>% pull(pct)),
    list(name = "Rest World", data = d %>% dplyr::filter(Q3 =="Other") %>% pull(pct))
  ) %>%
  hc_legend(
    align = "center",
    verticalAlign = "top",
    layout = "horizontal"
  )

Programming Language

d1 = kaggle_2021_compare %>%
  select(Q3, Q7_Part_1, Q7_Part_2) %>%
  group_by(Q3) %>%
  count(Q7_Part_1 == "Python") %>%
  mutate(pct  = round(n/sum(n)*100, digits = 1)) %>%
  na.omit()

d2 = kaggle_2021_compare %>%
  select(Q3, Q7_Part_2) %>%
  group_by(Q3) %>%
  count(Q7_Part_2 == "R") %>%
  mutate(pct  = round(n/sum(n)*100, digits = 1)) %>%
  na.omit()

d3 = kaggle_2021_compare %>%
  select(Q3, Q7_Part_8) %>%
  group_by(Q3) %>%
  count(Q7_Part_8 == "Julia") %>%
  mutate(pct  = round(n/sum(n)*100, digits = 1)) %>%
  na.omit()

Common Stacks

p1 = kaggle_2021_compare %>%
  select(Q3, starts_with("Q7")) %>%
  unite(Combined, starts_with("Q7"), sep = ",", na.rm = T) %>%
  count(Q3, Combined) %>%
  filter(Q3 == "Greece") %>%
  mutate(pct = round(n/sum(n)*100, digits = 1)) %>%
  arrange(-pct) %>%
  filter(pct >=3) %>%
  ggplot() +
  geom_col(aes(x = reorder(Combined, -pct), y = pct)) +
  geom_text(aes(x = Combined, y = pct+ 1, label = paste0(pct, "%", "", " (", n, ")"))) +
  labs(
    title = "Popular Stacks in Greece",
    subtitle = "Python seems to be THE tool for a big portion of Greek Analysts.",
    caption = "stesiam, 2023",
    x = "Stacks",
    y = "Percentage"
  ) + 
  theme_classic() +
  theme(
    plot.title = element_markdown(family = "title"),
    plot.subtitle = element_markdown(family = "subtitle", lineheight = 0.4),
    plot.caption = element_markdown(family = "subtitle"),
    text = element_text(family = "subtitle")
  )

kaggle_2021_compare %>%
  select(Q3, starts_with("Q7")) %>%
  unite(Combined, starts_with("Q7"), sep = ",", na.rm = T) %>%
  count(Q3, Combined) %>%
  filter(Q3 == "Other") %>%
  mutate(pct = round(n/sum(n)*100, digits = 1)) %>%
  arrange(-pct) %>%
  filter(pct >=3) %>%
  ggplot() +
  geom_col(aes(x = reorder(Combined, -pct), y = pct)) +
  geom_text(aes(x = Combined, y = pct+ 1, label = paste0(pct, "%", "", " (", n, ")"))) +
  labs(
    title = "Popular Stacks in Greece",
    subtitle = "Python seems to be THE tool for a big portion of Greek Analysts.",
    caption = "stesiam, 2023",
    x = "Stacks",
    y = "Percentage"
  ) + 
  theme_classic() +
  theme(
    plot.title = element_markdown(family = "title"),
    plot.subtitle = element_markdown(family = "subtitle", lineheight = 0.4),
    plot.caption = element_markdown(family = "subtitle"),
    text = element_text(family = "subtitle")
  )

p2 = kaggle_2021_compare %>%
  select(Q3, starts_with("Q7")) %>%
  unite(Combined, starts_with("Q7"), sep = ",", na.rm = T) %>%
  count(Q3, Combined) %>%
  filter(Q3 == "Other") %>%
  mutate(pct = round(n/sum(n)*100, digits = 1)) %>%
  arrange(-pct) %>%
  filter(pct >=3) %>%
  ggplot() +
  geom_col(aes(x = reorder(Combined, -pct), y = pct)) +
  geom_text(aes(x = Combined, y = pct+ 1, label = paste0(pct, "%", "", " (", n, ")"))) +
  labs(
    title = "Popular Stacks in Greece",
    subtitle = "Python seems to be THE tool for a big portion of Greek Analysts.",
    caption = "stesiam, 2023",
    x = "Stacks",
    y = "Percentage"
  ) + 
  theme_classic() +
  theme(
    plot.title = element_markdown(family = "title"),
    plot.subtitle = element_markdown(family = "subtitle", lineheight = 0.4),
    plot.caption = element_markdown(family = "subtitle"),
    text = element_text(family = "subtitle")
  )

IDEs

Python users

R users

Jobs

jobs1 = kaggle_2021_compare %>%
  select(Q3, Q5) %>%
  filter(Q3 == "Greece") %>%
  count(Q5) %>%
  arrange(-n) %>%
  head(8) %>%
  mutate(pct = round(n/sum(n)*100, 1))

jobs2 = kaggle_2021_compare %>%
  select(Q3, Q5) %>%
  filter(Q3 == "Other") %>%
  count(Q5) %>%
  arrange(-n) %>%
  head()  %>%
  mutate(pct = round(n/sum(n)*100, 1))

Conclusions

Acknowledgements

Dataset based on 2021 Kaggle Machine Learning & Data Science Survey

Image by Christina Smith from Pixabay

References

Arel-Bundock, V. (2025). Countrycode: Convert country names and country codes. Retrieved from https://vincentarelbundock.github.io/countrycode/

Arel-Bundock, V., Enevoldsen, N., & Yetman, C. (2018). Countrycode: An r package to convert country names and country codes. Journal of Open Source Software, 3(28), 848. Retrieved from https://doi.org/10.21105/joss.00848

Auguie, B. (2017). gridExtra: Miscellaneous functions for "grid" graphics. https://doi.org/10.32614/CRAN.package.gridExtra

Auguie, B., Goldie, J., & Thériault, R. (2023). Ggflags: Plot flags of the world in ggplot2. Retrieved from https://github.com/jimjam-slam/ggflags

Cuilla, K. (2022). Reactablefmtr: Streamlined table styling and formatting for reactable. Retrieved from https://kcuilla.github.io/reactablefmtr/

Heads or Tails. (2020). Kaggle has 10 million registered users! Retrieved from https://www.kaggle.com/discussions/general/332147

Hester, J., & Bryan, J. (2024). Glue: Interpreted string literals. Retrieved from https://glue.tidyverse.org/

Iannone, R. (2024). Fontawesome: Easily work with font awesome icons. Retrieved from https://github.com/rstudio/fontawesome

Kunst, J. (2022). Highcharter: A wrapper for the highcharts library. Retrieved from https://jkunst.com/highcharter/

Lin, G. (2025). Reactable: Interactive data tables for r. Retrieved from https://glin.github.io/reactable/

O’Hara-Wild, M. (2018, July 10). Arranging Hex Stickers in R. Retrieved from https://mitchelloharawild.github.io/mitchelloharawild.com//blog/hexwall

Qiu, Y., & See file AUTHORS for details., authors/contributors of the included fonts. (2020). Showtextdb: Font files for the showtext package. https://doi.org/10.32614/CRAN.package.showtextdb

Qiu, Y., & See file AUTHORS for details., authors/contributors of the included fonts. (2024a). Sysfonts: Loading fonts into r. Retrieved from https://github.com/yixuan/sysfonts

Qiu, Y., & See file AUTHORS for details., authors/contributors of the included software. (2024b). Showtext: Using fonts more easily in r graphs. Retrieved from https://github.com/yixuan/showtext

R Core Team. (2025). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/

The World Bank. (2023). Labor force participation rate (. Retrieved from https://genderdata.worldbank.org/indicators/sl-tlf-acti-zs/?geos=WLD&view=trend

Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag New York. Retrieved from https://ggplot2.tidyverse.org

Wickham, H. (2025). Forcats: Tools for working with categorical variables (factors). Retrieved from https://forcats.tidyverse.org/

Wickham, H., Chang, W., Henry, L., Pedersen, T. L., Takahashi, K., Wilke, C., … van den Brand, T. (2025). ggplot2: Create elegant data visualisations using the grammar of graphics. Retrieved from https://ggplot2.tidyverse.org

Wickham, H., François, R., Henry, L., Müller, K., & Vaughan, D. (2023). Dplyr: A grammar of data manipulation. Retrieved from https://dplyr.tidyverse.org

Wickham, H., Hester, J., & Bryan, J. (2025). Readr: Read rectangular text data. Retrieved from https://readr.tidyverse.org

Wickham, H., Vaughan, D., & Girlich, M. (2025). Tidyr: Tidy messy data. Retrieved from https://tidyr.tidyverse.org

Wilke, C. O., & Wiernik, B. M. (2022). Ggtext: Improved text rendering support for ggplot2. Retrieved from https://wilkelab.org/ggtext/

Zhu, H. (2024). kableExtra: Construct complex table with kable and pipe syntax. Retrieved from http://haozhu233.github.io/kableExtra/

Citation

BibTeX citation:

@online{2023,
  author = {, stesiam},
  title = {Kaggle’s {Greek} {Community}},
  date = {2023-05-06},
  url = {https://stesiam.com/posts/kaggle-greek-community/},
  langid = {en}
}

For attribution, please cite this work as:

stesiam. (2023, May 6). Kaggle’s Greek Community. Retrieved from https://stesiam.com/posts/kaggle-greek-community/

Predict Possible Interested Clients

stesiam — Thu, 24 Nov 2022 00:00:00 GMT

Introduction

On this article, I will make a ML model, in order to predict the costumers of a bank that are interested on having a term deposit. To achieve this goal I will use some Boosting algorithms (XGBoost and LightGBM). The data is from UCI website (Dua & Graff, 2017) and the dataset that I worked with is “Bank Marketing” (Moro, Cortez, & Rita, 2014).

But first let’s explain some things.

What is a term deposit ?

Generally, we are saving our money in a bank account but we are promising to not withdraw them for a specific amount of time.

And why someone to do that ?

It is apparent that term deposits have a major disadvantage over regular ones in terms of capital availability. Of course, people will not have motive to do that without anything in return. For that reason, the banks usually offer higher interest for that kind of accounts. For example, Piraeus Bank (a bank in Greece) offers double interest rate on term deposits compared to regular ones.

So, we can assume that there will be a serious interest for that product mostly on people with significant amount of savings () and do not have large, extraordinary expenses (loan, kids, etc.).

Prerequisites

Import libraries

For this analysis we will need standard libraries for importing and processing my data, such as readr (Wickham, Hester, & Bryan, 2025) and dplyr (Wickham, François, Henry, Müller, & Vaughan, 2023). The kableExtra (Zhu, 2024) package was used to print the results in table format. Concerning general purpose R libraries, I also used gridExtra in order to show ggplot items side to side.

On its basis this analysis is about to predict if someone is interested or not to have a term deposit. Thus, we need to build a ML model. The all-in-one solution package tidymodels is crucial to this. Although, there are some concerns about our data (imbalanced predicted value and our implementation of LightGBM. Thankfully, these are solved by bonsai and treesnip packages, respectively.

Finally, the ggplot2 (Wickham, Chang, et al., 2025) package was used to create some visualizations, as well as an auxiliary package, ggtext (R-ggtext?), for further formatting those.

Show the code

Install LightGBM and CatBoost on Ubuntu 22.04

stesiam — Sun, 13 Nov 2022 00:00:00 GMT

Introduction

When someone starts with Machine Learning he usually starts to build some simple models as logistic regression, naive Bayes, linear regression etc. And those alone are already enough for most use cases, as their simplicity is productivity-friendly and comes up with adequate accuracy. However, in enterprise level, accuracy can be important for a lot of reasons. Gradient Boosting Machines are some algorithms which outperform the aforementioned methods and are not complex enough to use them. Of course, before we build the model with (e.g. tidymodels) we have to install them. Thus, on this article I gather all that information.

Table 1: Documentation - Installation of GBMs

Installation Guides	Source
LightGBM	Link
CatBoost	Link
XGBoost	Link

Show the code

Git Configuration

stesiam — Fri, 04 Nov 2022 00:00:00 GMT

Introduction

What is Version Control System

A Version Control System (VCS) is a way to manage and track code changes. As we build an application, we add functions, change frontend features, fix bugs. We will need to edit the code many times. So we need a way to manage these changes. The tracking of every change to our code is the key point of VCS.

Workflow without Version Control

Some popular Version Control Software are the following :

Git
Apache Subversion (SVN)
Mercurial
Bazaar

Popularity of VCS

Having seen what a VCS is, it would be interesting to see which one is the most widely used. In order to study which VCS is the most popular, I pulled data from Google Trends.

Show the code

Forecasting Unemployment in Greece

stesiam — Sat, 22 Oct 2022 00:00:00 GMT

Introduction

Background

Time series

Methodology

Short Answer

If you are in a hurry, I predicted that unemployment on Greece is expected to further reduce. It will range between 10% - 13% in February, 2023.

Prerequisites

Import Libraries

For this analysis we will need standard libraries for importing and processing my data, such as readr (Wickham, Hester, & Bryan, 2025) and dplyr (Wickham, François, Henry, Müller, & Vaughan, 2023). The kableExtra (Zhu, 2024) package was used to print the results in table format, while the flextable (Gohel & Skintzos, 2025) package was used to print the results of the Dickey-Fuller and KPSS tests.

Then, due to the nature of the data (time series) it was deemed necessary to use relevant libraries such as lubridate(Spinu, Grolemund, & Wickham, 2024), tseries(Trapletti & Hornik, 2026) & forecast(R. Hyndman et al., 2026) packages.

Finally, the ggplot2 (R-ggplot2?) package was used to create some visualizations, as well as an auxiliary package, ggtext (R-ggtext?), for further formatting those.

Show the code

EDA on Greek Parliament

stesiam — Mon, 10 Oct 2022 00:00:00 GMT

Introduction

And here we go…

This is the first notebook on my website and I’d like to be a little special. I didn’t want to just take a ready-made dataset and apply a machine learning technique (I will do this in the next articles). So, I decided to make my own with the help of hellenic’s parliament website.

The Greek political scene has particularly preoccupied global public opinion in recent years, due to the Greek economic crisis. A part of it was spent on the reasons that caused it. The causes of the Greek crisis are many and a point of contention to this day. In this article we will deal essentially with one of the points of criticism. The election of the same persons.

With this notebook I will try to analyze whether this claim is valid by counting how many times someone has been elected to the Greek parliament. In addition, I will study the obsession with the same persons at the party level, but also at the local level (constituencies).

Prerequisites

Import Libraries

First and foremost, we have to load our libraries.

Show the code

List of Quarto Websites

stesiam — Wed, 10 Aug 2022 00:00:00 GMT

Introduction

Note

I have decided to also maintain this list as a project on the corresponding GitHub Repo page

At the time of writing, Quarto has surpassed other website-building packages in popularity (3.4k stars), compared to packages like blogdown (1.7k ) and distill (400 ). Given Quarto’s growing popularity, I took the initiative to compile a list of websites built with it. This way, data analysts who use other site-building methods (e.g., distill, blogdown) will have the chance to see what a Quarto website can do. Additionally, this catalogue may also help those who have already added Quarto to their toolkit, as they can draw inspiration from a centralised collection of relevant examples.

Quarto logo

Unfortunately, as is probably already obvious, this list cannot possibly include every Quarto website. For the most part, these are sites I have come across and liked. If you know of another site, or even your own, feel free to leave a comment or send me a message and I will add it as soon as possible.

Inspiration

A few years earlier I decided to build a website so I would have a single place to showcase my projects rather than having them scattered across various repositories on GitHub. So I tried several solutions that needed to support Rmd files (R Markdown files). I ended up using distill, and I genuinely loved it, as the website combined simplicity with elements that made it look quite professional. Despite whatever issues it had (many of which were later resolved by Quarto), I was quite satisfied. However, at that time I was a beginner programmer with fairly limited knowledge of HTML and CSS. This meant I could only make minimal modifications, and even those would take me a very long time. During the same period I discovered a website that catalogued Distill websites. That list helped me:

Distill logo

Understand the capabilities and limitations of distill package
Find motivation to get started by drawing inspiration from other people’s sites
Easily access interesting site features alongside their corresponding source code

In 2022, Quarto was announced and the landscape of writing and publishing articles, reports, and websites changed considerably. Several Distill users decided to make the switch and are already using Quarto for their personal websites (based on the Distill website list). Some may still be hesitant about this change, since Quarto is fairly new and there may be significant issues that haven’t been resolved yet. Another factor is the time required to learn how to use it, given that its documentation is particularly extensive. This is precisely why I believe this list is important: to make Quarto more accessible to new users and to encourage existing users to adopt new features.

blogdown logo

List of Quarto Sites

An indicative list of websites using Quarto is as follows:

Table 1: List of Quarto Websites

User	Website URL	Repository
aeturell	aeturrell.com	home
alexpghayes	alexpghayes.com	quarto-blog
almeidasilvaf	almeidasilvaf.github.io/	almeidasilvaf.github.io
andreashandel	andreashandel.com/	andreashandelwebsite
andrewheiss	andrewheiss.com	ath-quarto
andrewheiss	nonprofitf22.classes.andrewheiss.com	nonprofitf22.classes.andrewheiss.com
aster-hu	asterhu.com/	Asteroid_Blog
beatrizmilz	beamilz.com	blog-en
Bioconductor	bioconductor.github.io/biocblog/	biocblog
cgoo4	quantumjitter.com/	quantumjitter
CrumpLab	crumplab.com/	CrumpLab.github.io
currocam	currocam.github.io/biocblog/	currocam.github.io
cynthiahqy	cynthiahqy.com/	digital-garden
daxkellie	daxkellie.com	website-quarto
ddimmery	ddimmery.com	quarto-website
djnavarro	blog.djnavarro.net/	quarto-blog
drganghe	drganghe.github.io	drganghe.github.io
ekholme	ericekholm.com/	ee-quarto-site
EllaKaye	ellakaye.co.uk/	ellakaye.co.uk
EmilHvitfeldt	emilhvitfeldt.com	emilhvitfeldt.com
epiforecasts	epiforecasts.io/	epiforecasts.github.io
fusionet24	myyearindata.com/	myyearindata
ivelasq	ivelasq.rbind.io	pipedream
JavOrraca	javierorracadeatcu.com/	quarto-blog
jbkunst	jkunst.com/blog/	blog
jessesadler	jessesadler.com	quarto-blog
jeweljohnsonj	sciquest.netlify.app//	SciQuest
jhelvy	jhelvy.com	jhelvy_quarto
joelnitta	joelnitta.com	joelnitta-home
journalovi	journalovi.org	journalovi.github.io
jthomasmock	TheMockup.blog	themockup-blog
kathoffman	khstats.com/	khstats-quarto
kelly-sovacool	sovacool.dev//	kelly-sovacool.github.io
kurianbenoy	kurianbenoy.com	kurianbenoy-website
magsol	magsol.github.io/	magsol.github.io
marioangst	marioangst.com/en/	–
markusschanta	blog.markus.schanta.at/	blog
marvinschmitt	marvinschmitt.com/	marvinschmitt-dot-com
matherion	behaviorchange.eu/	personal-website
maxdrohde	maximilianrohde.com	blog_quarto
mcanouil	mickael.canouil.fr/	mickael.canouil.fr
mdsumner	hypertidy.org	quarto-blog
Meghansaha	thetidytrekker.com/	thetidytrekker-quarto
mine-cetinkaya-rundel	A Quarto tip a day	quarto-tip-a-day
miriamheiss	miriamheiss.com/	miriam-blog
mmhamdy	hypothesis-space.netlify.app/	Hypothesis-Space
njlyon0	njlyon0.github.io	njlyon0.github.io
nucleic-acid	jollydata.blog	quarto-blog
numbats	numbat.space/	numbats-quarto-website
Openscapes	openscapes.github.io/quarto-website-tutorial/	quarto-website-tutorial
pat-alt	paltmeyer.com/	pat-alt.github.io
paul-buerkner	paul-buerkner.github.io/	paul-buerkner.github.io
pkollenda	philippkollenda.com/	Website
Possible-Institute	possible.institute	website
quarto-dev	quarto.org	quarto-web
rlbarter	rebeccabarter.com	personal-website-quarto
robertmitchellv	robertmitchellv.com	robertmitchellv.github.io
robjhyndman	robjhyndman.com/	robjhyndman.com
rsangole	rsangole.netlify.app/	blog
samanthacsik	samanthacsik.github.io/	samanthacsik.github.io
seeM	wasimlorgat.com	blog
shamindras	shamindras.com	ss_personal_distill_blog
srvanderplas	srvanderplas.github.io/	srvanderplas.github.io
StefanThoma	stefanthoma.github.io/quarto-blog/	quarto-blog
stesiam	stesiam.com	stesiam.github.io
tidymodels	tidymodels.org	tidymodels.org
laderast	laderast.github.io/	laderast.github.io
vbaliga	vbaliga.github.io	vbaliga.github.io
willingc	willingconsulting.com/	willing-consulting-2022
zekiakyol	zekiakyol.com/	personal-website

Multilingual Websites

Unfortunately, Quarto does not natively support the creation of multilingual websites. This can be problematic for those who wish to publish their content in their native language as well, which may differ from English. For now, the solution officially offered by Quarto is through project profiles, but the implementation seems quite complex to me and is barely covered in the relevant documentation for this particular use case. If you are interested in developments regarding multilingual support in Quarto, you can check out the discussion on this topic on the project’s GitHub page. As of now, Quarto does not officially support multilingual websites, e-books, and so on in a straightforward manner.

Λογότυπο του ROpenSci

ROpenSci is a Non-Profit Organisation founded in 2011. The organisation promotes a culture of collaboration in research, where data is openly available and software can be reused, so that research and its results are reliable, transparent, and easily reproducible. ROpenSci has created many important packages for the R language in order to advance its vision, many of which are vital to the R analyst community. Notable examples include babelquarto, which enables the creation of multilingual sites with Quarto, and {targets}, which helps organise code execution pipelines. For a full list of packages created and maintained by ROpenSci, you can click here

Despite the lack of native support, the babelquarto R package was created to address this problem and make building multilingual sites more feasible and user-friendly.Some websites have already integrated babelquarto into their workflow and now offer content in multiple languages - one of which is the site you are currently reading. A great example of a multilingual (babelquarto) website (japanese / english) is Joel Nitta’s.

Although there is no official support from Quarto, babelquarto was created, an R package that attempts to solve this problem and make the creation of multilingual sites both possible and easy. Several sites have already integrated babelquarto into their workflow and now offer content in multiple languages. One of them is the very website you are reading right now! You can find its source code here. Another nice example of a multilingual site (Japanese / English) is that of Joel Nitta and its corresponding repository.

I hope you found all of this interesting and, why not, that it convinced you to build your own Quarto website.

Citation

BibTeX citation:

@online{2022,
  author = {, stesiam},
  title = {List of {Quarto} {Websites}},
  date = {2022-08-10},
  url = {https://stesiam.com/posts/list-of-quarto-sites/},
  langid = {en}
}

For attribution, please cite this work as:

stesiam. 2022. “List of Quarto Websites.” August 10, 2022. https://stesiam.com/posts/list-of-quarto-sites/.

Hello, Quarto

stesiam — Wed, 27 Jul 2022 00:00:00 GMT

Hello, Quarto !

I have finally built my website using Quarto. Currently, the page is hosted via GitHub Pages.

How it all started

My encounter with Statistics was rather accidental. The first time I sat for the Panhellenic exams, I was admitted to the Department of Economics at the University of Peloponnese. Although I liked the idea of studying Economics, the prospect of relocating to another city (Tripoli) posed an enormous financial burden accompanied by great uncertainty, something I simply couldn’t afford. So, I chose to retake the Panhellenic exams the very next year. From what I remember, I had listed Economics and Computer Science departments first, followed by Statistics departments, prioritising those in the Attica region and then the rest of Greece. In the end, when the admission scores were announced, I found out I had been admitted to the Department of Statistics and Insurance Science at the University of Piraeus. Being, then, a student in a Statistics department, I began gaining my first relevant exposure through the university. During the first years of my studies I focused on theory in the fields of Statistics, Probability, Insurance, and Actuarial Science. This gave me a solid theoretical foundation. However, I was missing the most important trait for a Statistician: the ability to put my knowledge into practice and apply the corresponding statistical techniques in real-world settings. The university placed very little emphasis on this aspect. To illustrate, my year’s curriculum included only two courses on applied statistics using statistical software, namely Excel and SPSS. The SPSS course was decent, but it was by no means commensurate with the knowledge I had acquired, nor did it meet the modern demands of the job market. On top of that, SPSS as a piece of software has significant drawbacks that make it a poor choice, including the inability to customise your model, the lack of transparency in how results are produced, and of course its cost. During 2018-2019 I started looking for alternatives, as I absolutely loathed the software, or, to be more precise, the fact that I would have to pay for it. Among other things, I found free alternatives that followed the same logic as SPSS, such as PSPP, JASP, and jamovi. Personally, I consider the latter two to be perfectly respectable for simple analyses. In fact, I would recommend JASP and jamovi to anyone as a worthy alternative if they lacked programming skills and had no access to SPSS. It was around that time that I discovered there is a programming language focused on data analysis, R (also known by the hashtag #rstats).

My student ID from the Statistics department in Piraeus. I may no longer look like the kid in the photo, but this is indeed my ID. The kid in that picture doesn’t know it yet, but some serious storms are heading his way in a few years, though things will get better (hopefully).

From SPSS to R

Our university, like many universities around the world, offers its students free licences for various software. The problem is that the subscription expires as soon as you graduate.

Transitioning from a statistical suite to a programming language can be a traumatic experience, especially if it’s your first programming language. SPSS and its free alternatives accompanied me for a respectable period of time; however, the very defining strength of these applications gradually began turning into a weakness. These programs are designed so that individuals with basic knowledge and understanding of Statistics can carry out their own analyses. As I progressed through my studies, I kept learning more methodologies and techniques, until at some point I realised that these “ready-made” tools could only take me so far. They are undoubtedly useful, but they impose a ceiling on how deeply the user needs, or is allowed, to think. For anything beyond that ceiling, they either ignore it entirely because it’s considered too complex for the majority, or they adopt reasonable assumptions based on the data so that the average user doesn’t have to deal with theory or hypothesis testing. Where exactly that ceiling lies varies from application to application: some choose to be extremely simple and do a few things well for the everyday user (e.g., Excel), while others offer more options with greater complexity (e.g., SPSS, JASP, Weka). All of them, however, have their own limitations. As I worked with such software, I kept wondering what I would do if I wanted to set different parameters or apply a variation of a test I had read about in a recently published paper. That’s when I realised that, by continuing down this path, I would end up serving the software rather than my own purpose. My goal was clearly to master my Science, not a program or a statistical software suite. So, I decided to make the big switch.

First steps

My relationship with R began in 2018, with quite a few ups and downs. I can say that at first it felt peculiar, and for the most part I used it almost mechanically. In short, I was copying code and simply trying to understand what I was doing. To expose myself even further: if I wanted to check whether differences were statistically significant, I would Google it. Due to its idiosyncrasies as a language, I used it more as an analysis tool than as a programming language. My first uses of R involved some basic statistical tests, and I was impressed that I could now use a tool with theoretically unlimited capabilities compared to the “ready-made” programs. The whole journey, of course, was particularly interesting and involved many late nights. Despite all that, after countless error messages, I started to notice my gradual improvement. And I still hadn’t grasped what a good decision I had made, as I was unaware of the full extent of its capabilities. Over time, I discovered that you could create your own plots with the {ggplot2} package, and I was thrilled (admittedly a bit more than normal) when I learned that the BBC and the Financial Times use {ggplot2} to produce their impressive graphics. I then learned about other capabilities as well, the most notable being the creation of web applications and dashboards with the {Shiny} package, where the range of possibilities extended well beyond Statistics. Around the same time, I discovered Kaggle, a website dedicated to data science. I found it to be an incredibly useful platform, and that’s where I started publishing various notebook-articles, charts, and generally experimenting with R. Looking back today, after several years, my charts were abysmal, but back then I would look at them and feel proud. At least over time I improved, and I now have the self-awareness to admit that they were indeed terrible 😀. If you ask nicely, I might even upload one or two so you can see how clueless I was back then and we can all have a good laugh. Kaggle was extremely valuable. I learned a tremendous amount, and there were people working at major corporations who published solutions to real-world problems. To this day, competitions are posted and many share their solutions. The only problem is that Python gradually took over, and as a result the vast majority of content now covers only Python. This caused the R user community to shrink, with fewer and fewer R articles being published or being replaced by Python equivalents. Because of that, I made an interesting decision…

The R language is a programming language that specializes in data analysis and statistics. It was created in 1993 by Ross Ihaka and Robert Gentleman.

Out of platform

A few years ago I decided to build my own website so I could host my articles myself. First, I had to decide how to create it. There were many options, and I experimented with various approaches and tools. One solution would have been a conventional website built with WordPress, but that would have been costly since I would need to host it on a server and cover the associated expenses (roughly $10/month) for a decent and secure VPS plan. That wasn’t worth it. Being the cheapskate that I am, I looked for a more affordable way, and as it turns out, there is one. During that period I discovered static websites and the fact that a blog doesn’t actually need a server. Examples include Hugo with the help of the {blogdown} package and {distill}, which were decent solutions but came with significant issues. On the one hand, Hugo is built on a language I don’t know (Go), so if I wanted to make changes or add new functionality it would have been impossible. Moreover, my previous site, built with Distill, wasn’t mobile-friendly (non-responsive), as it would hide the menu button on small screens. Quarto solved many of the problems these two approaches had, and nowadays a great many people use it for their websites. Yes, Quarto still has some shortcomings compared to other established site-building frameworks (e.g., Astro), but right now it’s the best option for anyone working in analysis languages (R, Python, Julia) who wants to publish articles based on their work. Its main advantage, in my opinion, is the ability to execute an analysis file and export it directly as an article. If I were using another static site builder like Astro, I would have to run my code separately each time, paste the results or charts into the site, and rebuild, which would make the whole process unproductive.

(a) Distill logo

(b) Blogdown logo

Figure 1: R Packages for building websites with R (before Quarto)

Website goals

Since this is my first article on the website, I’d like to lay out my goals for it. So, here’s what I plan to do:

Write articles and carry out machine learning analyses (using R and Python)
Publish explanatory articles (e.g., “How to make your Quarto site” etc.)
Showcase my activities / projects (mainly my {Shiny} applications, etc.)
Publish material and articles in Greek, since there isn’t a large audience in Greece and I want that to change
Solve exam papers, or at least similar exercises, using R
And much more …, so stay tuned!

Acknowledgments

Image by R. E. Beck from Pixabay

References

Allaire, J. J., Teague, C., Scheidegger, C., Xie, Y., & Dervieux, C. (2022). Quarto. https://doi.org/10.5281/zenodo.5960048

Dervieux, C., Allaire, J., Iannone, R., Presmanes Hill, A., & Xie, Y. (2022). Distill: ’R markdown’ format for scientific and technical writing. Retrieved from https://CRAN.R-project.org/package=distill

Xie, Y., Dervieux, C., & Hill, A. P. (2022). Blogdown: Create blogs and websites with r markdown. Retrieved from https://github.com/rstudio/blogdown