Scala algorithm: Remove duplicates from an unsorted List

Published

Algorithm goal

Remove duplicates from an unsorted list; for example, [1,2,1,2,3] becomes [1,2,3] because the 1 can only be repeated once, the 2 can only be repeated once, and the 3 is only repeated once. Use cases here are especially in streaming and processing idempotent data, such as that in unreliable networks where packets or messages may be sent multiple times.

Test cases in Scala

This algorithm comes with test cases.
To see the free test cases for all our algos, as well as run them in our TDD IDE, Register (free).

Algorithm in Scala

17 lines of Scala (compatible versions 2.13 & 3.0), showing how concise Scala can be!

Get the full algorithm Scala algorithms logo, maze part, which looks quirky!

or

'Unlimited Scala Algorithms' gives you access to all the 89 published Scala Algorithms! Plenty to master your Scala algos skills!

Upon purchase, you will be able to Register an account to access all the algorithms on multiple devices.

Stripe logo

Explanation

Using a scanLeft and scanRight, we can do what would typically be a while-loop, however we do it in a declarative way where we check if the element has been seen, and if not, then we emit it as a Some(); otherwise we emit a None to say no value has been emmitted.

This is very close to a State machine, and in fact can be extracted out as such. See: RemoveDuplicatesFromSortedListStateMachine, ParenthesesFoldingStateMachine. (this is © from www.scala-algorithms.com)

Scala concepts & Hints

  1. Collect

    'collect' allows you to use Pattern Matching, to filter and map items.

    assert("Hello World".collect {
  case character if Character.isUpperCase(character) => character.toLower
} == "hw")

  2. Lazy List

    The 'LazyList' type (previously known as 'Stream' in Scala) is used to describe a potentially infinite list that evaluates only when necessary ('lazily').

  3. Option Type

    The 'Option' type is used to describe a computation that either has a result or does not. In Scala, you can 'chain' Option processing, combine with lists and other data structures. For example, you can also turn a pattern-match into a function that return an Option, and vice-versa!

    assert(Option(1).flatMap(x => Option(x + 2)) == Option(3))

assert(Option(1).flatMap(x => None) == None)

  4. Pattern Matching

    Pattern matching in Scala lets you quickly identify what you are looking for in a data, and also extract it.

    assert("Hello World".collect {
  case character if Character.isUpperCase(character) => character.toLower
} == "hw")

  5. scanLeft and scanRight

    Scala's `scan` functions enable you to do folds like foldLeft and foldRight, while collecting the intermediate results

    assert(List(1, 2, 3, 4, 5).scanLeft(0)(_ + _) == List(0, 1, 3, 6, 10, 15))

  6. Stack Safety

    Stack safety is present where a function cannot crash due to overflowing the limit of number of recursive calls.

    This function will work for n = 5, but will not work for n = 2000 (crash with java.lang.StackOverflowError) - however there is a way to fix it :-)

    In Scala Algorithms, we try to write the algorithms in a stack-safe way, where possible, so that when you use the algorithms, they will not crash on large inputs. However, stack-safe implementations are often more complex, and in some cases, overly complex, for the task at hand.

    def sum(from: Int, until: Int): Int =
  if (from == until) until else from + sum(from + 1, until)

def thisWillSucceed: Int = sum(1, 5)

def thisWillFail: Int = sum(1, 300)

Scala Algorithms: The most comprehensive library of algorithms in standard pure-functional Scala

Study our 89 Scala Algorithms: 6 fully free, 89 published & 0 upcoming

Fully unit-tested, with explanations and relevant concepts; new algorithms published about once a week.

  1. Compute the length of longest valid parentheses
  2. Check a binary tree is balanced
  3. Make a queue using stacks (Lists in Scala)
  4. Find height of binary tree
  5. Single-elimination tournament tree
  6. Reverse Polish Notation calculator
  7. Quick Sort sorting algorithm in pure immutable Scala
  8. Find minimum missing positive number in a sequence
  9. Least-recently used cache (LRU)
  10. Count pairs of a given expected sum
  11. Compute a Roman numeral for an Integer, and vice-versa
  12. Compute keypad possibilities
  13. Matching parentheses algorithm with foldLeft and a state machine
  14. Traverse a tree Breadth-First, immutably
  15. Read a matrix as a spiral
  16. Remove duplicates from a sorted list (state machine)
  17. Token Bucket Rate Limiter
  18. Leaky Bucket Rate Limiter
  19. Merge Sort: stack-safe, tail-recursive, in pure immutable Scala, N-way
  20. Longest increasing sub-sequence length
  21. Reverse first n elements of a queue
  22. Binary search a generic Array
  23. Game of Life
  24. Merge Sort: in pure immutable Scala
  25. Make a queue using Maps
  26. Is an Array a permutation?
  27. Count number of contiguous countries by colors
  28. Add numbers without using addition (plus sign)
  29. Tic Tac Toe MinMax solve
  30. Run-length encoding (RLE) Encoder
  31. Print Alphabet Diamond
  32. Find kth largest element in a List
  33. Balanced parentheses algorithm with tail-call recursion optimisation
  34. Reverse a String's words efficiently
  35. Count number of changes (manipulations) needed to make an anagram with an efficient foldLeft
  36. Count passing cars
  37. Establish execution order from dependencies
  38. Counting inversions of a sequence (array) using a Merge Sort
  39. Longest common prefix of strings
  40. Check if an array is a palindrome
  41. Compute missing ranges
  42. Check a directed graph has a routing between two nodes (depth-first search)
  43. Compute nth row of Pascal's triangle
  44. Run-length encoding (RLE) Decoder
  45. Check if a number is a palindrome
  46. In a range of numbers, count the numbers divisible by a specific integer
  47. Compute minimum number of Fibonacci numbers to reach sum
  48. Find the index of a substring ('indexOf')
  49. Reshape a matrix
  50. Compute the steps to transform an anagram only using swaps
  51. Compute modulo of an exponent without exponentiation
  52. Closest pair of coordinates in a 2D plane
  53. Find the contiguous slice with the minimum average
  54. Compute maximum sum of subarray (Kadane's algorithm)
  55. Pure-functional double linked list
  56. Binary search in a rotated sorted array
  57. Check if a directed graph has cycles
  58. Rotate Array right in pure-functional Scala - using an unusual immutable efficient approach
  59. Check a binary tree is a search tree
  60. Length of the longest common substring
  61. Sliding Window Rate Limiter
  62. Tic Tac Toe board check
  63. Find an unpaired number in an array
  64. Check if a String is a palindrome
  65. Count binary gap size of a number using tail recursion
  66. Remove duplicates from a sorted list (Sliding)
  67. Monitor success rate of a process that may fail
  68. Least-recently used cache (MRU)
  69. Find sub-array with the maximum sum
  70. Find the minimum absolute difference of two partitions
  71. Find maximum potential profit from an array of stock price
  72. Fibonacci in purely functional immutable Scala
  73. Fizz Buzz in purely functional immutable Scala
  74. Find triplets that sum to a target ('3Sum')
  75. Find combinations adding up to N (non-unique)
  76. Find the minimum item in a rotated sorted array
  77. Make a binary search tree (Red-Black tree)
  78. Remove duplicates from an unsorted List
  79. Mars Rover
  80. Find combinations adding up to N (unique)
  81. Find indices of tuples that sum to a target (Two Sum)
  82. Count factors/divisors of an integer
  83. Compute single-digit sum of digits
  84. Fixed Window Rate Limiter
  85. Traverse a tree Depth-First
  86. Reverse bits of an integer
  87. Find k closest elements to a value in a sorted Array
  88. QuickSelect Selection Algorithm (kth smallest item/order statistic)
  89. Rotate a matrix by 90 degrees clockwise

Explore the 21 most useful Scala concepts

To save you going through various tutorials, we cherry-picked the most useful Scala concepts in a consistent form.

  1. Class Inside Class
  2. Class Inside Def
  3. Collect
  4. Def Inside Def
  5. Drop, Take, dropRight, takeRight
  6. foldLeft and foldRight
  7. For-comprehension
  8. Lazy List
  9. Option Type
  10. Ordering
  11. Partial Function
  12. Pattern Matching
  13. Range
  14. scanLeft and scanRight
  15. Sliding / Sliding Window
  16. Stack Safety
  17. State machine
  18. Tail Recursion
  19. Type Class
  20. View
  21. Zip

Register now (free)

 Register with GitHub    

This gives you access to free test cases & hints for all our 89 published algorithms as well as our installation-free Scala IDE.

You can then choose to subscribe to "Scala Algorithms Unlimited", which gets you access to every current and upcoming algorithm solution. We use Stripe so your data is safe.

We will e-mail you at most once per week with new algorithms and relevant content.
Your data is protected and unsubscribing is straightforward.

Stripe logo

No GitHub? please visit our Membership page

How the algorithms look

  1. A description/goal of the algorithm.
  2. An explanation with both Scala and logical parts.
  3. A proof or a derivation, where appropriate.
  4. Links to Scala concepts used in this specific algorithm, also unit-tested.
  5. An implementation in pure-functional immutable Scala, with efficiency in mind (for most algorithms, this is for paid subscribers only).
  6. Unit tests, with a button to run them immediately in our in-browser IDE.
Screenshot of an example algorithm demonstrating the listed features